Package: detprocess
detprocess: Detector processing code for feature extractiondetprocessis a Python package meant for feature extraction from raw detector data saved bypytesdaq. The main functionality of the code is contained indetprocess.process, which contains all the possible features to be extracted and the general pipeline of how features are extracted. This package also contains helper IO functions for loading events frompytesdaqand saving the processed data as Vaex DataFrames.Table of ContentsInstallationUsageAvailable Base FeaturesYAML FileExtract featuresInstallationTo install the most recent release ofdetprocess, type the following line into your command linepip install detprocess --upgradeTo install the most recent development version ofdetprocess, clone this repo, then from the top-level directory of the repo, type the following line into your command linepip install .If using a shared Python installation, you may want to add the--userflag to the above line. Note the package requirements, especially the need forQETpyandpytesdaq.UsageOne of the goals of this package is to keep the feature extraction pipeline simple and modular. The pipeline in mind can be approximated as follows:Know what features you want to extract, see:Available FeaturesCreate a YAML file specifying feature extraction options, see:YAML FileRun the feature extraction code on your data, see:Extracting FeaturesAvailable Base FeaturesThe available features to extract are stored as the static methods ofdetprocess.FeatureExtractors. Each of these methods take your data and extract that specific feature from each event.At this time, the available features are:of1x1_nodelay: returns the no delay optimum filter amplitude and chi-square (as in, the amplitude if the template is not allowed a time degree-of-freedom)of1x1_unconstrained: returns the unconstrained optimum filter amplitude, time offset, and chi-squareof1x1_constrained: returns the constrained optimum filter amplitude, time offset, and chi-square, where a window constraint is specifiedbaseline: returns the average value from the beginning of an event up to some specified indexintegral: returns the integral of the event(no baseline subtraction)from some specified start index to some specified end indexmaximum: returns the maximum value of the event from some specified start index to some specified end indexminimum: returns the minimum value of the event from some specified start index to some specified end indexenergyabsorbed: returns the energy absorbed by a Transition-Edge Sensor (TES) based on the inputted parameters that correspond to the TES bias pointThe base features can be used to define new features in the configuration with different settings, for example "baseline_pre" defined in a yaml file (see below) can use the "baseline" based algorithm.More features can be added either locally, or if there is a feature that is universally useful, then please submit a Pull Request!There are also features that are stored directly inpytesdaqfiles, whichdetprocesswill also extract. These are:event_numberevent_indexdumpn_umberseries_numberevent_timetrigger_typetrigger_amplitudetrigger_timeTo understand these more, we direct the user topytesdaq's Documentation.YAML FileThe YAML file contains nearly all of the information needed to extract features from your data. This is done on purpose, as it allows the user to easily reuse/change their YAML files for different processing, to easily version control their processing, and easily share their processing setup with collaborators. To make sure we can do this, the YAML must  have a specific format. Here's an example below.filter_file:./filter_example.hdf5detector1:o1x1_nodelay:run:Trueof1x1_unconstrained:run:Falseof1x1_constrained:run:Falsewindow_min_from_trig_usec:-400window_max_from_trig_usec:400of1x1_constrained_glitchrun:Truewindow_min_from_trig_usec:-400window_max_from_trig_usec:400base_algorithm:of1x1_unconstrainedtemplate_tag:glitchbaseline_pre:run:Truebase_algorithm:baselinewindow_min_from_start_usec:0window_max_from_trig_usec:-1000integral:run:Truestart_index:0window_min_from_trig_usec:-500window_max_from_trig_usec:500In this YAML file, we first specify the filter file, which contains the PSD and templates for each channels. The pulse template should be a single array that contains the expected pulse shape, normalized to have a pulse amplitude of 1 and have a baseline of 0. The current-referenced PSD should be a single array that contains the two-sided PSD in units of $\mathrm{A}^2/\mathrm{Hz}$. Note that both of these will should have the same digitization rate and/or length as the data that will be processed to be able to calculate the optimum filter features. We must then specify which channel will be processed, in this casedetector1. This should match the channel name in the correspondingpytesdaqfile. the optimum filter features.We have also specified to extract different features from each event:of1x1_nodelay,baseline, andintegral. This is done by specifyingrun: Truein the file, as compared torun: Falseforof1x1_unconstrainedandof1x1_constrained. Note that it is fine to simple exclude features from the YAML file, as they simply will not be calculated (e.g.energyabsorbedis not included in this example).Extracting FeaturesSee notebook detprocess/examples/run_detprocess.ipynb