Package: wipac-file-catalog-indexer
file-catalog-indexerIndexing package and scripts for the File CatalogHow ToAPIfrom indexer.index import indexThe flagship indexing functionFind files rooted at given path(s), compute their metadata, and upload it to File CatalogConfigurable for multi-processing (default: 1 process) and recursive file-traversing (default: on)Internally communicates asynchronously with File CatalogNote: Symbolic links are never followed.Note:index()runs the current event loop (asyncio.get_event_loop().run_until_complete())Ex:index(index_config,# see config.py for a description of the fields in these typed dictionariesoauth_config,rest_config)from indexer.index import index_fileCompute metadata of a single file, and upload it to File Catalog, i.e. index one fileSingle-processed, single-threadedawaitindex_file(filepath='/data/exp/IceCube/2018/filtered/level2/0820/Run00131410_74/Level2_IC86.2018_data_Run00131410_Subrun00000000_00000172.i3.zst',manager=MetadataManager(...),fc_rc=RestClient(...),)from indexer.index import index_pathsA wrapper aroundindex_file()which indexes multiple files, and returns any nested sub-directoriesSingle-processed, single-threadedNote: Symbolic links are never followed.sub_dirs=awaitindex_paths(paths=['/data/exp/IceCube/2018/filtered/level2/0820','/data/exp/IceCube/2018/filtered/level2/0825'],manager=MetadataManager(...),fc_rc=RestClient(...),)from indexer.metadata_manager import MetadataManagerThe internal brain of the Indexer. This has minimal guardrails, does not communicate to File Catalog, and does not traverse file directory tree.Metadata is produced for an individual file, at a time.Ex:manager=MetadataManager(...)# caches connections & directory info, manages metadata collectionmetadata_file=manager.new_file(filepath)# returns an instance (computationally light)metadata=metadata_file.generate()# returns a dict (computationally intense)Scriptspython -m indexer.indexA command-line alternative to usingfrom indexer.index import indexUse with-hto see usage.Note: Symbolic links are never followed.python -m indexer.generateLikepython -m indexer.index, but prints (usingpprint) the metadata instead of posting to File Catalog.Simply, uses file-traversing logic around calls toindexer.metadata_manager.MetadataManagerNote: Symbolic links are never followed.python -m indexer.delocateFind files rooted at given path(s); for each, remove the matching location entry from its File Catalog record.Note: Symbolic links are never followed..i3 File Processing-Level Detection and Embedded Filename-Metadata ExtractionRegex is used heavily to detect the processing level of a.i3file, and extract any embedded metadata in the filename. The exact process depends on the type of data:Real Data (/data/exp/*)This is a two-stage process (seeMetadataManager._new_file_real()):Processing-Level Detection (Base Pattern Screening)The filename is applied to multiple generic patterns to detect if it is L2, PFFilt, PFDST, or PFRawIf the filename does not trigger a match,only basic metadata is collected(logical_name,checksum,file_size,locations, andcreate_date)Embedded Filename-Metadata ExtractionAfter the processing level is known, the filename is parsed using one of (possibly) several tokenizing regex patterns for the best match (greedy matching)If the filename does not trigger a match,the function will raise an exception (script will exit).This probably indicates that a new pattern needs to be added to the list.seeindexer.metadata.real.filename_patternsSimulation Data (/data/sim/*)This is a three-stage process (seeMetadataManager._new_file_simulation()):Base Pattern ScreeningThe filename is checked for.i3file extensions:.i3,.i3.gz,.i3.bz2,.i3.zstIf the filename does not trigger a match,only basic metadata is collected(logical_name,checksum,file_size,locations, andcreate_date)there are a couple hard-coded "anti-patterns" used for rejecting known false-positives (see code)Embedded Filename-Metadata ExtractionThe filename is parsed using one of MANY (around a thousand) tokenizing regex patterns for the best match (greedy matching)If the filename does not trigger a match,the function will raise an exception (script will exit).This probably indicates that a new pattern needs to be added to the list.seeindexer.metadata.sim.filename_patternsProcessing-Level DetectionThe filename is parsed for substrings corresponding to a processing levelseeDataSimI3FileMetadata.figure_processing_level()If there is no match,processing_levelwill be set toNone, since the processing level is less important for simulation data.Metadata SchemaSee:Google DocFile Catalog TypesWarningsRe-indexing Files is Tricky (Two Scenarios)Indexing files that have not changed locations is okay--this probably means that the rest of the metadata has also not changed. A guardrail query will check if the file exists in the FC with thatlocationsentry, and will not process the file further.HOWEVER, don't point the indexer at restored files (of the same file-version)--those that had their initiallocationsentry removed (ie. removed from WIPAC, then moved back). Unlike re-indexing an unchanged file, this file will befully locally processed(opened, read, and check-summed) before encountering the checksum-conflict then aborting. These files will be skipped (not sent to FC), unless you use--patch(replaces thelocationslist, wholesale), which isDANGEROUS.Example Conflict: It's possible a file-version exists in FC after initial guardrailsfile was at WIPAC & indexedthen moved to NERSC (locationadded) & deleted from WIPAC (locationremoved)file was brought back to WIPACnow is being re-indexed at WIPACCONFLICT -> has the samelogical_name+checksum.sha512but differinglocations