Package: servicex-databinder
ServiceX DataBinderRelease v0.5.0servicex-databinderis a user-analysis data management package using a single configuration file.
Samples with external data sources (e.g.RucioDIDorXRootDFiles) utilize ServiceX to deliver user-selected columns with optional row filtering.The following table shows supported ServiceX transformers by DataBinderInput formatCode generatorTransformerOutput formatROOT Ntuplefunc-adluprootrootorparquetATLAS Release 21 xAODfunc-adlatlasr21rootROOT Ntuplepython functionpythonrootorparquetPrerequisiteAccess to a ServiceX instancePython 3.7+Installationpipinstallservicex-databinderConfiguration fileThe configuration file is a yaml file containing all the information.Thefollowing example configuration filecontains minimal fields. You can also downloadservicex-opendata.yamlfile (rename toservicex.yaml) at your working directory, and run DataBinder for OpenData without an access token.General:ServiceXName:servicex-opendataOutputFormat:parquetSample:-Name:ggH125_ZZ4lepXRootDFiles:"root://eospublic.cern.ch//eos/opendata/atlas/OutreachDatasets\/2020-01-22/4lep/MC/mc_345060.ggH125_ZZ4lep.4lep.root"Tree:miniColumns:lep_pt, lep_etaGeneralblock requires two mandatory options (ServiceXNameandOutputFormat) as in the example above.Input dataset for each Sample can be defined either byRucioDIDorXRootDFilesorLocalPath.ServiceX query can be constructed with either TCut syntax or func-adl.Options for TCut syntax:Filter1andColumnsOption for Func-adl expression:FuncADL1Filterworks only for scalar-type of TBranch.Output format can be eitherApache parquetorROOT ntupleforuprootbackend. OnlyROOT ntupleformat is supported forxAODbackend.The followings are available options:Option forGeneralblockDescriptionDataTypeServiceXName*ServiceX backend name in yourservicex.yamlfileStringOutputFormat*Output file format of ServiceX delivered data (parquetorrootforuproot/rootforxaod)StringTransformerSet transformer for all Samples. Overwrites the default transformer in theservicex.yamlfile.StringDeliveryDelivery option;LocalPath(default) orLocalCacheorObjectStoreStringOutputDirectoryPath to a directory for ServiceX delivered filesStringWriteOutputDictName of an ouput yaml file containing Python nested dictionary of output file paths (located in theOutputDirectory)StringIgnoreServiceXCacheIgnore the existing ServiceX cache and force to make ServiceX requestsBoolean*Mandatory optionsOption forSampleblockDescriptionDataTypeNameSample name defined by a userStringTransformerTransformer for the given sampleStringRucioDIDRucio Dataset Id (DID) for a given sample;Can be multiple DIDs separated by commaStringXRootDFilesXRootD files (e.g.root://) for a given sample;Can be multiple files separated by commaStringTreeName of the input ROOTTTree;Can be multipleTTrees separated by comma (uprootONLY)StringFilterSelection in the TCut syntax, e.g.jet_pt > 10e3 && jet_eta < 2.0(TCut ONLY)StringColumnsList of columns (or branches) to be delivered; multiple columns separately by comma (TCut ONLY)StringFuncADLFunc-adl expression for a given sampleStringLocalPathFile path directly from local path (NO ServiceX tranformation)StringA config file can be simplified by utilizingDefinitionblock. You can define placeholders underDefinitionblock, which will replace all matched placeholders in the values ofSampleblock. Note that placeholders must start withDEF_.You can source each Sample using different ServiceX transformers.
The default transformer is set bytypeofservicex.yaml, butTransformerin theGeneralblock overwrites if present, andTransformerin eachSampleoverwrites any previous transformer selection.Thefollowing example configurationshows how to use each Options.General:ServiceXName:servicex-uc-afTransformer:uprootOutputFormat:rootOutputDirectory:/Users/kchoi/data_for_MLstudyWriteOutputDict:fileset_ml_studyIgnoreServiceXCache:FalseSample:-Name:SignalRucioDID:user.kchoi:user.kchoi.signalA,user.kchoi:user.kchoi.signalB,user.kchoi:user.kchoi.signalCTree:nominalFuncADL:DEF_ttH_nominal_query-Name:Background1XRootDFiles:DEF_ggH_inputTree:miniFilter:lep_n>2Columns:lep_pt, lep_eta-Name:Background2Transformer:atlasr21RucioDID:DEF_Zee_inputFuncADL:DEF_Zee_query-Name:Background3LocalPath:/Users/kchoi/Work/data/background3-Name:Background4Transformer:pythonRucioDID:user.kchoi:user.kchoi.background4Function:|def run_query(input_filenames=None):import awkward as ak, uproottree_name = "nominal"o = uproot.lazy({input_filenames:tree_name})return {"nominal: o}Definition:DEF_ttH_nominal_query:"Where(lambdae:e.met_met>150e3).\Select(lambdaevent:{'el_pt':event.el_pt,'jet_e':event.jet_e,\'jet_pt':event.jet_pt,'met_met':event.met_met})"DEF_ggH_input:"root://eospublic.cern.ch//eos/opendata/atlas/OutreachDatasets\/2020-01-22/4lep/MC/mc_345060.ggH125_ZZ4lep.4lep.root"DEF_Zee_input:"mc15_13TeV:mc15_13TeV.361106.PowhegPythia8EvtGen_AZNLOCTEQ6L1_Zee.\merge.DAOD_STDM3.e3601_s2576_s2132_r6630_r6264_p2363_tid05630052_00"DEF_Zee_query:"SelectMany('lambdae:e.Jets(\"AntiKt4EMTopoJets\")').\Where('lambdaj:(j.pt()/1000)>30').\Select('lambdaj:j.pt()/1000.0').\AsROOTTTree('junk.root','my_tree',[\"JetPt\"])"Deliver datafromservicex_databinderimportDataBindersx_db=DataBinder('<CONFIG>.yml')out=sx_db.deliver()The functiondeliver()returns a Python nested dictionary that contains delivered files.Input configuration can be also passed in a form of a Python dictionary.Delivered Samples and files in theOutputDirectoryare always synced with the DataBinder config file.Error handlingfailed_requests=sx_db.get_failed_requests()If failed ServiceX request(s),deliver()will print number of failed requests and the name of Sample, Tree if present, and input dataset. You can get a full list of failed samples and error messages for each byget_failed_requests()function. If it is not clear from the message you can browseLogsin the ServiceX instance webpage for the detail.Useful toolsCreate Rucio container for multiple DIDsThe current ServiceX generates one request per Rucio DID.
It's often the case that a physics analysis needs to process hundreds of DIDs.
In such cases, the script (scripts/create_rucio_container.py) can be used to create one Rucio container per Sample from a yaml file.
An example yaml file (scripts/rucio_dids_example.yaml) is included.Here is the usage of the script:usage:create_rucio_containers.py[-h][--dry-runDRY_RUN]infilecontainer_nameversion

CreateRuciocontainersfrommultipleDIDs

positionalarguments:infileyamlfilecontainsRucioDIDsforeachSamplecontainer_namee.g.user.kchoi:user.kchoi.<container-name>.Sample.v1versione.g.user.kchoi:user.kchoi.fcnc_ana.Sample.<version>

optionalarguments:-h,--helpshowthishelpmessageandexit--dry-runDRY_RUNRunwithoutcreatingnewRuciocontainerAcknowledgementsSupport for this work was provided by the the U.S. Department of Energy, Office of High Energy Physics under Grant No. DE-SC0007890