Package: reprox
ReProx: ReProcessing for XENONnTPackageCIDocumentationPlease visitthe documentationfor installation instructions and examples.ExamplesCan be found eitheron githubor theonline documentation.Reprocessing on daliProcess data in so far available on dali with the current containerLogicThere are several (sequential) steps with (associated scripts):Step 1. Find runs to process (reprox-find-data)Step 2. Process the runs that were found (reprox-start-jobs)Step 3. Move the data that was processed to the desired folder (reprox-move-folders)One can also run these three steps from one file (reprox-reprocess), which runs all three in
order.The best place to start is by going over these files and doreprox-find-data --helpto see which options there are. Most are discussed below.Running step by stepBelow, we show how these three steps are done. This can also be done in one
commandskip to single command.Step 0 - Activation and test installationYou only have to do it once, to prevent confusion we will go over it step by step.First, activate a container (NB! the singularity containers do not work
as they cannot communicate with the job submission of dali).source/cvmfs/xenon.opensciencegrid.org/releases/nT/development/setup.sh
gitclonegit@github.com:XENONnT/reprox.git
pipinstall-ereprox--usertest that the installation is complete and successfulreprox-find-data--helpTrouble-shootingNow, the commands above may sometimes not work as expected due to permission errors on the
containers. If there is an error, you could seereprox-find-data: command not found.
If this is the case, simply navigate to thebinfolder ofreproxand
run the commands as below:cdrepox/bin
pythonreprox-find-data--helpThe otherreproxscripts are similarly located in thebinfolder. If you had
to change this once, you have to dopython <script>for all the scripts
listed below.Step 1 - finding data to (re)process on daliNow we have to know which data to process, this can be done with the following
command. Determine which data to process:reprox-find-data\--packagecutax\--contextxenonnt_v6\--targetevent_infoevent_pattern_fitcuts_basic\--cmt-versionglobal_v6The--packageand--contextarguments specify where to load the context
from (straxen/cutax) and which context to use. In this example, we usexenonnt_v6.
The--targetargument specifies which datatypes to produce. This can be a
list as in the example above. We will check if the datatypes can be produced for this given context.
Since some context may use a global CMT version that is only valid for a range of runs,
the--cmt-versionis specified separately and tells the script to only process runs
that are valid in thiscmt_version. This can be disabled using--cmt-version False(for example, you know that the CMT version is always valid for the datatypes you requested).This takes a while (+/- 30 minutes) and writes a file
called/dali/lgrandi/xenonnt/data_management_reprocessing/to_do_runs.csv(depending on your ini
file). This file has a list of runs that you can process given the options as above.Step 2 - starting the jobs to process the dataAfter producing/dali/lgrandi/xenonnt/data_management_reprocessing/to_do_runs.csv, we need to
submit the jobs to process the data. Most of the arguments are the same as above,
we now also specify some self-explanatory arguments for the jobs to be submitted.reprox-start-jobs\--packagecutax\--contextxenonnt_v6\--targetevent_infoevent_pattern_fitcuts_basic\--ram12000\--cpu2Step 3 - move to the production folderNow, hopefully most of the data has been processed successfully, we can now move it to the
production folder. This includes a check to see if the data was processed successfully so
even if a few jobs failed (or are still running), you can safely run this command below.reprox-move-foldersRun entire workflow (steps 1-3 in a single command)You can also do all the above in a single command, using the same arguments (see above for explanation of each.).reprox-reprocess\--packagecutax\--contextxenonnt_v6\--targetevent_infoevent_pattern_fitcuts_basic\--cmt-versionglobal_v6\--ram12000\--cpu2\--move-after-workflow# To move the data into the production folderAdvanced usageBelow are several more advanced use cases.Changing the defaults of processingYou might want to play with the config file that says how many resources one uses by default.
Thereprocessing.inifile. You can either change the source code of this file, or you can overwrite it as follows:gitclonegit@github.com:XENONnT/reprox.git
cpreprox/reprox/reprocessing.inimy_reprocessing_config.ini# # Edit my_reprocessing_config.ini. For example using vim:# vi my_reprocessing_config.ini# overwrite the file used using an environment variableexportREPROX_CONFIG=$(pwd)/my_reprocessing_config.iniYou will see that your defaults have been changed (e.g. doreprox-reprocess --help) reflecting the
changes you made in the.inifile.Use custom configYou might want to process some data with slightly different settings, this can be done using
the--context_kwargsargument as follows
(please don't move it into the production folder unless you know what you are doing):reprox-reprocess\--packagecutax\--contextxenonnt_v6\--targetevent_infoevent_pattern_fitcuts_basic\--cmt-versionglobal_v6\--ram12000\--cpu2\--context-kwargs'{"s1_min_coincidence": 2, "s2_min_pmts": 10}'Usingreproxfrom your jupyter notebookYou can also run the commands from above in a notebook or python script.fromreproximportfind_data,submit_jobs,validate_runtargets='event_info event_pattern_fit cuts_basic'.split()# First determine which data to processfind_data.find_data(targets=targets,exclude_from_invalid_cmt_version='global_v6')# Now start running the jobssubmit_jobs.submit_jobs(targets=targets)# Finally move the jobs to the production foldervalidate_run.move_all()Processing NV dataBy default, the package assumes that only linked-mode or TPC runs are processed, if you want to
instead process NV data you need to tell the scripts to also take into account the NV detector:reprox-reprocess\--packagecutax\--contextxenonnt_v6\--targetevents_nv\--detectorsneutron_vetomuon_veto--ram12000\--cpu2\--cmt-versionFalseUsing tagged versionsOne might want to run with a different tag as soMY_TAG=2021.12.2source/cvmfs/xenon.opensciencegrid.org/releases/nT/$MY_TAG/setup.sh
reprox-reprocess\--packagecutax\--contextxenonnt_v5\--targetsevent_info\--cmt-versionglobal_v5\--ram24000\--cpu2\--move-after-workflow\--tag$MY_TAG0.2.1 / 2022-03-29Fix kwargs setting (#53)0.2.0 / 2022-03-29Fix deprecation warning (#50)Add timeout to jobs (#51)print reprox version (#52)Fix permissions (#49)Update coverage (#47)Run straxen version on coveralls (#48)Versions (#44, #45, #33, #43)0.1.0 / 2022-01-27Finetuning after testing (#18)requirement updates (#22)0.0.3 / 2022-01-17Bugfix data finding (#17)0.0.2 / 2022-01-17Bugfix for kwargs in job submssion (#15Update documentation (#13, fb0cbdab46f92d87871baae298efa20e45168e6d, 55144dc8c00094e9f64af0bc45f85bebe2b3b1e5,adf34cc4a494ad761a709edb1a83597a594ca238, 5e492ea8a9d1db3e426210b90c2eb2147eb214d9)Refactored by Sourcery (#14)Update with py3.10 (#16)Release (#12)0.0.1 / 2022-01-17Update documentation (#7, #11, 2895ebb5063f7951550c8e57059fdd3bbf600b9e)Add tests (#1, #9, #10)Pin requirements (#3, #4, #5, #6, #8)Release (#12)0.0.0 / 2022-01-17Initial commits