Package: utilix
utilixutilixis a utility package for XENON software, mainly relating to analysis. It currently has two main features: (1) a general XENON configuration framework and (2) easy access to the runsDB by wrapping python calls to a RESTful API. Eventually, we would like to include easy functions for interacting with the Midway and OSG batch queues.Installationgit clonethis repo and:# at top level of utilix
pip install -e ./ --userConfiguration fileThis tool expects a configuration file given by the environment variableXENON_CONFIG, defaulting to$HOME/.xenon_configif it is empty. Note that
environment variables can be used in the form$HOME. Example:[RunDB]
rundb_api_url = [ask teamA]
rundb_api_user = [ask teamA]
rundb_api_password = [ask teamA]The idea is that analysts could use this single config for multiple purposes/analyses.
You just need to add a (unique) section for your own purpose and then you can use theutilix.Configeasily. For example, if you made a new section calledWIMPwithdetected = yesunder it:from utilix.config import Config
cfg = Config()
value = cfg.get('WIMP', 'detected') # value = 'yes'For more information, see theConfigParserdocumentation, from whichutilix.config.Configinherits.Runs DatabaseNearly every analysis requires access to the XENON runsDB. The goal of utilix is to simplify the usage of this resource as much as possible. Therundbmodule includes two ways to access the runsDB:A RESTful API: a Flask app running at Chicago that queries the runDB in a controlled manner. This is the recommended way to query the database if the specific query is supported. The source code for this app can be foundhere.A wrapper aroundpymongo, which sets up the Mongo client for you, similarly to how we did queries in XENON1T. In that case each package usually needed its own copy + pasted boilerplate code; that code is now just included in utilix where it can be easily imported by other packages.RunDB APIRunDB API AuthenticationThe API authenticates using a token system.utilixmakes the creation and renewal of these tokens easy with theutilix.rundb.Tokenclass. When you specify a user/password in your utilix configuration file, as shown above, a token is saved locally at~/.dbtokenthat contains this information. This token is used/renewed as needed, depending on the users specified in the config file.Different API users have different permissions, with the general analysis user only able to read from the runDB and not write. This is an additional layer of security around the RunDB.Setting up the runDBThe goal of utilix is to make access to the runDB trivial. If using the runDB API, all you need to do to setup the runDB in your local script/shell isfrom utilix import dbThis instantiates the RunDB class, allowing for easy queries. Below we go through some examples of the type of queries currently supported by the runDB API wrapper in utilix.If there is functionality missing that you think would be useful, please contact teamA or make a new issue (or even better, a pull request).Query for runs by sourceNote that the interface returns pages of 1,000 entries, with the first page being 1.from utilix import db
data = db.query_by_source('neutron_generator', page_num=1)Get a full documentYou can also grab the full run document using the run number. A run name is also supported (from XENON1T days),
but not going to be used for XENONnTdoc = db.get_doc(7200)Get only the data entry of a documentdata = db.get_data(2000)Strax(en) ContextsIn XENONnT we need to track the hash (or lineage) that specifies a configuration for each datatype. We keep that information in a specific collection of the runDB. We can access that collection using the runDB API as shown below.For a given context name and straxen version, we can get the hash for each dataype. For example, for the xenonnt_online context and straxen version 0.11.0:>>> db.get_context('xenonnt_online', '0.11.0')

{'_id': '5f89f588d33cced1fd104ea5',
 'date_added': '2020-10-16T19:33:28.913000',
 'hashes': {'aqmon_hits': '4gwju6gdto',
            'corrected_areas': 'wbgyvcbq7i',
            'distinct_channels': 'a6orjqoffa',
            'energy_estimates': 'plxyjbnnui',
            'event_basics': 'pjql7a36yb',
            'event_info': 's4mprxr7qq',
            'event_info_double': 'hodh2726fi',
            'event_positions': 'tzzwhaf4gy',
            'events': 'aycpwrvco2',
            'hitlets_nv': 'pd2fuwzbjt',
            'led_calibration': 'lsdigsccxn',
            'lone_hits': 'nagx3zzuiv',
            'lone_raw_record_statistics_nv': 'vx2cbuxcxo',
            'lone_raw_records_nv': 'vx2cbuxcxo',
            'merged_s2s': '274qjtnjto',
            'merged_s2s_he': 'n5tib6rljj',
            'peak_basics': '2gq3vm6b2t',
            'peak_basics_he': 'ucjr5lvvbg',
            'peak_positions': 'c7vzetbnqw',
            'peak_proximity': '7nvjls2mab',
            'peaklet_classification': 'gpnp6dzxc4',
            'peaklet_classification_he': 'tmlx5dkfcf',
            'peaklets': 'nagx3zzuiv',
            'peaklets_he': 'mu26cr25vf',
            'peaks': 'dmavalropc',
            'peaks_he': 'hzamxeoer6',
            'pulse_counts': 'jxkqp76kam',
            'pulse_counts_he': '5mepav2zzf',
            'raw_records': 'rfzvpzj4mf',
            'raw_records_aqmon': 'rfzvpzj4mf',
            'raw_records_aqmon_nv': 'rfzvpzj4mf',
            'raw_records_coin_nv': 'vx2cbuxcxo',
            'raw_records_he': 'rfzvpzj4mf',
            'raw_records_mv': 'rfzvpzj4mf',
            'raw_records_nv': 'rfzvpzj4mf',
            'records': 'jxkqp76kam',
            'records_he': '5mepav2zzf',
            'records_nv': 'btmwmjvgdp',
            'veto_intervals': '7q35r23nba',
            'veto_regions': 'jxkqp76kam'},
 'name': 'xenonnt_online',
 'strax_version': '0.12.2',
 'straxen_version': '0.11.0'}If you know the specific datatype whose hash you need, use insteadget_hash:>>> db.get_hash('xenonnt_online', 'peaklets',  '0.11.0')
'nagx3zzuiv'If you are deemed worthy to have write permissions to the runDB (you have a corresponding user/password with write access in your config file), you can also add documents to the context collection with>>> db.update_context_collection(document_data)wheredocument_datais a dictionary that contains the context name, straxen version, hash information, and more as shown in the example above.Boilerplate pymongo setupThe runDB API is the recommended option for most database queries, but sometimes a specific query isn't supported or you might want to do complex aggregations, etc. For that reason,utilixalso includes a wrapper aroundpymongoto setup the MongoClient. To use this, you need to specify in your config file[RunDB]
pymongo_url = [ask someone]
pymongo_database = [ask someone]
pymongo_user = [ask someone]
pymongo_password =  [ask someone]Note that this is needed in addition to the runDB API fields. Given the correct user/password, you can setup the XENONnT runDB collection for queries as follows:>>> from utilix.rundb import pymongo_collection
>>> collection = pymongo_collection()Then you can query the runDB using normal pymongo commands. For example:>>> collection.find_one({'number': 9000}, {'number': 1})
{'_id': ObjectId('5f2d999448350bff030d2d3b'), 'number': 9000}You can also access different collections by passing an argument topymongo_collection. The analogous query to the contexts collection shown above would be:>>> collection = pymongo_collection('contexts')
>>> collection.find_one({'name': 'xenonnt_online', 'straxen_version': '0.11.0'})If you need to use different databases or do not want to use the information listed in your utilix configuration file, you can also pass keyword arguments to overwrite the information in the config file. This is useful if you need to e.g. use both XENONnT and XENON1T collections. To use the XENON1T collection, for example:>>> xe1t_coll, xe1t_db, xe1t_user, xe1t_pw, xe1t_url = [ask someone]
>>> xe1t_collection = pymongo_collection(xe1t_coll, database=xe1t_coll, user=xe1t_user, password=xe1t_pw, url=xe1t_url)Data processing requestsYou may find yourself missing some data which requires a large amount of resources to process. In these cases, you can submit a processing request to the computing team.If you have utilix installed, your context will have therequest_processingmethod.Example:st=straxen.test_utils.nt_test_context()requests=st.request_processing('012882',# can be a tuple of run_ids'event_basics',# must be a single data typepriority=1,# higher values mean higher priority, defaults to -1comments='this is for the new elife analysis',# add some comments to help people reviewing the requests.submit=True,#If you use `submit=False` the requests objects will be created but no submitted.)The script will open a web-browser if possible, and also print out the URL you need to visit to authorize the script to operate on your behalf. After you login using github and authorize the request, the script will get a token and save it in memory for the next requests.TODOWe want to implement functionality for easy job submission to the Midway batch queue.
Eventually we want to do the same for OSG.It would be nice to port e.g. the admix database wrapper to utilix, which can then be used
easily by all analysts.v0.5.3 - 2021/04/14Small bugfixes in utilix.iov0.5.2 - 2021/04/13Actually fixes the bug introduced in v0.5.0.v0.5.1 - 2021/04/12Fixes the bug introduced in v0.5.0 for cases when no config file exists. (#41, #42)v0.5.0 - 2021/04/12GridFS interface for both direct mongo access and through the API (#35).Allow users to bypass the auto-initialization of the DB class with a config option (#39)Hopefully a fix to the annoying error that requires removing the dbtoken (#40)And a few other small bug fixes / typos.v0.4.1 - 2021/02/03Minor changes:Don't fail on import (https://github.com/XENONnT/utilix/pull/33)Add version number to utilix, add bumpversion (https://github.com/XENONnT/utilix/pull/31)v0.4.0 - 2021/01/21Major updates:Different collections for XENON1T and XENONnT (#29)Minor updatesAdd methods for MC collection (#30)Bug fixes (#24, #25)Windows support (#26)Better error handling (#27)