Metadata: Conditions data Handling Detector Characterization Data (Metadata) in XENONnT Conditions Data Processed Data Metadata Storage As an example, in XENONnT we needed to track crucial detector conditions like liquid xenon purity, which we monitor in terms of the electron lifetime. The electron lifetime determines the likelihood of electron absorption by impurities, thereby affecting energy reconstruction of interactions. How we calculate this value has changed over time so we needed to keep track of these multiple methods via different versions. Red: 222Rn alpha, Blue: 83mKr, green: 37Ar. Including the condition by charge up Electric ﬁeld model. Gray band is the measurement by purity monitor, with systematic uncertainty. RFrame is a metadata framework that offers indexing features, similar to Pandas dataframes, but can operate on multiple storage backends. The goal of RFrame is to provide the following properties: ● Support for multi-indexing using discrete or continuous values, allowing for complex data structures to be easily represented. ● Checking and enforcement of values and index data types to ensure data integrity and consistency. ● Arbitrary rules for reading and writing to the database, giving developers ﬂexibility in how they interact with the data. ● Support multiple data access storages. ● Python functions can be used to insert, delete, and modify data, allowing for easy integration with existing codebases. Xedocs was built based on RFrame and it handles the XENONnT speciﬁc needs for metadata management. At its core, xedocs consists of a collection of schemas that deﬁne the rules that different kinds of metadata need to follow. Base corrections Time sampled Time interval Base resource reference Map corrections Example: Electron Lifetime By: Luis Sanchez, Rice University, USA Yossi Mosbacher, Weizmann institute of science, Israel Aaron Higuera, Rice University, USA Christopher Tunnell, Rice University, USA RFrame Xedocs JSON Mongo Tinydb Our processing pipeline uses global versions, which is a list of lineages specifying the version of all the conditions metadata used to process a set of data. The conditions metadata can be separated into 2 broad categories: ● ONLINE version: Updated periodically and allows for extrapolation of values. Values here can only be changed in the future in terms of the time index. ● OFFLINE version: Allows for interpolation when processing a data segment with values before and after the time the data was taken. Ofﬂine versions can become obsolete as they are replaced with new versions. Raw Data Processing Pipeline Experiment Links To code: ●https://github.com/jmosbacher/rframe ●https://github.com/XENONnT/xedocs Experiment Monitoring Tools Manual Inputs - Version indexing - Time indexing - Updating rules - Other conditions metadata dependent requirements xarray API ODM API Has its own versioning system Pandas df Lists Dicts - _Alias attribute is always unique - Schemas include a versioned index - Changing already set values is not allowed - Adds an interpolation index of type datetime - Enforces rules on inserting new data points - Adds an interval index of type datetime. - Enforces rules on updating intervals. - Requires existence of resource in a database prior to inserting a new document - Requires an ‘algorithm’ index of type string. These schemas can be used to deﬁne the different kinds of conditions metadata we need and are built inheriting for the previous one. API’s Data backends In memory datatypes Many experiments have their own system to store, organize and track conditions data, but in many cases it is not possible to copy the system of another experiment due to how specialized to their experiment this system will be. In XENONnT we developed an open source tool in python to facilitate conditions data management and storage, which we call rframe and xedocs. We needed tools to handle versioned and time-indexed conditions data, as well as to impose rules regarding how the va