Package: libDetXR
Detector X-Ray Python Packagemost of these python scripts can be called with option ‘-h’ to get more detailed help.libDetXR.pywrapper to the libDetXR.(dll,so) library. These contains following C-functions:Compression AlgorithmsImaging AlgorithmsBitManip Functionsfor details s.a.:
C/C++:../src/README.rstSPEC2hdf5.pyInput:*.cbf files and *.dat filesOutput:one *.hdf5 fileSpeedup:muptiprocess single nodecollects data of a scan on the cSAXS beamline and stores it in one HDF5 fileThis will analyse and merge following data into one hdf5 file:specES1/dat-files/specES1_started_2013_04_26_1604.dat
mcs/S00000-00999/S00033/e14472_00033.dat
eiger/S00000-00999/S00033/e14472_1_00033_00000_00001.cbf and many more
pilatus_1/S00000-00999/S00033/e14472_1_00033_00000_00001.cbf and many more
pilatus_2/S00000-00999/S00033/e14472_1_00033_00000_00001.cbf and many morethis can be speedup on one compute node using all available cores.ImgSource.pyImgSource is a helper class to provide image. It is used among others in zmqImageSource, procMoment, procRoiStat, procSTXM
Currently following sources are supported:Random values:
  --src rand    X    Y Z --type=DATATYPE
  --src rand 2560 2160 3 --type=uint16
Incremental values:
  --src inc    X    Y Z --type=uint16
  --src inc 2560 2160 6 --type=uint16
Raw data from files *.raw in a directory:
  --src raw    X    Y PATH                       --type=DATATYPE
  --src raw 2560 2160 /scratch/detectorData/PCO/ --type=uint16
TIFF data from files *.tif[f] in a directory:
  --src tif PATH
  --src tif /scratch/detectorData/PCO/
CBF data from files *.cbf in a directory:
  --src cbf PATH
  --src cbf /scratch/detectorData/e14472/pilatus_1/S00000-00999/S00033/
HDF file from a given object:
  --src hdf FILE ELEM
  --src hdf myfile.hdf5 entry/mygrp/dataset
ZMQ source stream (now only uncompressed 'chunk'-type is supported)
  --src zmq JSON-config (with server,optional (queueSz, ifType and timeout))
  --src zmq '{"server":"tcp://localhost:8080","queueSz":4,"ifType":"PULL"}',zmqImageSource.pyZeroMQ PUSH or PUB Server:the datasource can be one of the types that ImgSource supports. It then sends this images as ZMQ messages (chunk type, raw or compressed).Input:a data source: *.cbf, *.tiff, *.hdf5 *.raw zmqStream etc. filesOutput:one zmq-Streams with ‘chunk’ header (raw or compressed)Speedup:only single processprocMoment.pyProcess the moments m00,m11,m01,m02,m10,m20 of an image source (cbf, tiff,raw files or hdf5 file) and stores the result in an output hdf5 file.Input:ImgSource.py data: (raw, tiff, cbf, hdf5, zmq-steam, etc.), optional MATLAB Mask file for valid pixelsOutput:one *.hdf5 file or a zmq StreamSpeedup:only single processFurther a visualization of the moments is possible.Different implementation can be taken: python/opencv or CMulti process speedup is not yet implemented but feasible on request.SIMD implementation would also speedup on request.Single image processing (32 bit, 1679x1475 pilatus_2 image, on pc 9477):
:python: 0.5sec
:pyFast: 0.03sec
:openCV: 0.005sec
:c:      0.005sec
The speed now is mostly memory bandwidth bound.procRoiStat.pyProcess statistics of multiple ROI of an image source (cbf, tiff,raw files or hdf5 file) and stores the result (avg,avgstd,sumsq) for each ROI in an output hdf5 file.Input:ImgSource.py data: (raw, tiff, cbf, hdf5, zmq-steam, etc.), and a MATLAB Roi fileOutput:one *.hdf5 file or a zmq StreamSpeedup:only single processThe input for ROI definition is a MATLAB Roi file (e.g. pilatus_integration_mask.mat) as used at the cSAXS beamline.Currently only the avg is implemented.The speed is much higher as the origin MATLAB implementation.Multi process speedup is not yet implemented but feasible on request.SIMD implementation would also speedup on request.Single image processing (32 bit, 1679x1475 pilatus_2 image, on pc 9477):The speedup depends on the mask. The mask used is a 16 segments pilatus_integration_mask.matSpeedup C-implementation vesus python: 41-42 x fasterThe python implementation is allready much faster than the original MATLAB.python:0.235 secc:0.006secTherefore we can achive a speedup of 100-500x compared to the MATLAB code.
The speed now is mostly memory bandwidth bound.procSTXM.pySTXM processing of an image source and visualization. The current implementation is very basic, just as a template to later have a faster and more flexible STXM processing script.Input:ImgSource.py data: (raw, tiff, cbf, hdf5,  zmq-steam, etc.), optional MATLAB Mask file for valid pixelsOutput:x,y,t during processing on console. STXM viewer and /tmp/result.npz file when processing is finished.Speedup:only single process, multiproc-single node for hdf5->hdf5 processingzmqWriter.pyZmqWriter connects to a ZMQ server that sends json- and binary data messages.Input:one or multiple zmq-Streams with ‘chunk’ or ‘pilatus’(cbf-files) headerOutput:one *.hdf5 fileSpeedup:muptiprocess single node (compression). Single process writing to file.These data is stored into a hdf5 file or it can just copy*.cbffiles.The program can also be started as a REST server .The REST-server and each writer is a independent process.Multi process compression speedup for the writer process is not foreseen, because the zmqWriter is intended to receive compressed chunks. Anyhow: the zmqWriter can convert images to a desired compression scheme, but this will work single threaded and could be therefore time critical:1. *.cbf-Files            -> *.cbf-Files
2. raw image              -> hdf5-File (uncompressed, byte-shuffle, zlib lz4, lzf)
3. *.cbf-Files            -> hdf5-File (uncompressed, byte-shuffle, zlib lz4, lzf)
4. compressed image-chunk -> hdf5-File (compressed as the received image-chunk)The modes used in productions are 1 for cbf files as from the Pilatus and 4 for hdf5 files with the Eiger.The modes 2 and 3 are for tests and preparation to hdf5 format and not recommanded for production. They are not performantCbf2CrystFELhdf5.pyconverts cbf files to CrystFEL hdf5 files.
the program searches all *.cbf files of a directory and converts them.
S.a.http://www.desy.de/~twhite/crystfel/Further helper classesutilities.py:utilities to create HDF5 objectsCbfParser.py:class to parse*.cbffiles. It gets the header and decompress the binary part to a numpy arrayDatParser.py:class to parse*.datfiles at the cSAXS beamline.zmq2imgGL.py:a zmq sink to show an show a colored image that is received.*.cbfzmq messages, uncompressed and compressed chunks are supported.FileNameGen.py:generate filenames with a given filename pattern. (Currently not used)hdf5vis.py:Simple Test application to show movies of image series in a hdf5-filelibDetXRTester.py:sample code to test the libDetXR functionality.testPerfHdf5ChunkWrite.py:Test the performance of Chunk writing:all source data is first read into ram, then a HDF5 file is written.diffferent interprocess communications are tested and compared:'Pipe1','Pipe1b','Pipe2','Pool1','ShMem1','ShMem2','ShMem3','ShPool2'