Package: bdrc-irat
Parallel scanThis folder contains utilities that survey the entire archive file system and/or s3, depending on the task,
collect, and aggregate data. It is limited to the hard coded roots of the archive (/mnt/Archive[0..n]) the BUDA (archive.tbrc.org/Works)It was written in python to avoid nasty file globbing issues having to do with reserved shell characters in file names,
and to exploit parallelism.Thanks to Élie Roux for providing the released work and image group lists that served as the input.The motivation was to have in one place a definition of graphics images, as ordinary file listing techniques left it up
to the researcher to filter out the counts. In an ideal world, we would test and count by using a graphics library to
open and taste each file (thescan-imagesactiontypesactually does this, to extract the image type from the
image file metadata), but this is hugely expensive in practice.The rationale behind only including image files is that other files don't really matter, and don't generally hurt BUDA
performance. Although updates since 2019 are clean (audit-tool), there's no real use of cleaning out old files that aren't bothering
anybody. As well, correspondences between S3 images/ folders and our file systems aren't perpetually guaranteed.Image files are included in input calculations with a regular expression incommon.pyThis is a one line change that
can update the scans as needed# common.py# re stringGRAPHICS_FILE_EXTS:str=r'.+\.(jpg|jpeg|tif|tiff|png|bmp|wmf|pdf)$'# Example action_list.py...img_re=re.compile(GRAPHICS_FILE_EXTS,re.IGNORECASE)...s3_images_list.extend([x['Key']forxinobject_listifimg_re.match(x['Key'])])Regexp was chosen overfnmatchdue to efficiency and being able to select case sensitivity (or not)InstallationUntil I build a pyPI installation, you can run manually.git clone|pullthis repository, and run directly from theparallelscanfolder.Python 3 is required, and some elements may require python 3.9 or later
Seerequirements.txtfor the python libraries you have to install. AO recommends using a venv.
We also strongly recommend installingwheelbefore the others (pip 23has gotten all rigid on us)UsageThe only executable in this folder isscan-images❯./scan-images--h
usage:Seeparallelscan/README.docfordetails

Runsimagescanningtoolsagainstasetofworks

optionalarguments:-h,--helpshowthishelpmessageandexit-a{list,types,sizes},--action{list,types,sizes}Availableactions-wWORK_RIDS[WORK_RIDS...],--work_ridsWORK_RIDS[WORK_RIDS...]oneormorework_rids-iINPUT_LIST_FILE,--input_list_fileINPUT_LIST_FILEfilecontaininglistofwork_ridsorpaths(see-c)-oOUTPUT_FILE,--output_fileOUTPUT_FILE[Optional]outputfile(defaultstdout)Lyingargparsesays the-a/--actionargument is optional, it is not.File arguments-w/--work_ridsJust what it appears to be.-i/--input-filelist of entities to search. this is Work RIDs only.-a/--actionthe three possible actions, or modes. These are documented below.ActionslistThis action counts image files by image group and emits four columns in a csv (see `published_work_file_counts.csv)
Where the image group could not be found, blank columns are emitted. Where the image group was found, but contained no images,
the count is shown as 0.OutputsThe following sections describe the output of each of the actions:typeslistsizestypesThis output was the original instance of the queue pattern. (seebuda-base/archive-ops#549)
The output is a list of individual files whose file extension does not match the image type
as PIL sees it.listReturns counts, by image groups of file system (archive) and s3 (web)work,ig,n_fs,n_s3
W00CHZ0103341,I1CZ35,210,210
W00EGS1016181,I1PD10388,183,183
W933,I5700,225,0
W933,5700,0,225
WEAP039-1-4-130,,,
WEAP039-1-4-140,,,
WEAP039-1-4-150,,,
WEAP039-1-4-160,,,Takes about 3 hours for the whole oeuvre. (71611 image groups)[EDT08/23/2318.41.41]:root:{p-00}-DEBUG-collectedcountW9140sec:29.049865[EDT08/23/2318.41.41]:root:{p-06}-DEBUG-collectedcountW8LS31064sec:401.735611[EDT08/23/2318.41.41]:root:{MainThread}-INFO-Donewaiting[EDT08/23/2318.41.41]:root:{MainThread}-INFO-<EndET:12805.011623

~/prod/ao927----took3h33m26sat18:41:42sizesThesizesaction aggregates all thepublished images graphics files (children of theimages/directory - assumes all image groups are published)other graphic files under the workand emits a csv file with their counts.work,non-image-size,non-image-count,image-size,image-count
W00EGS1016242,36827083,6,31098214,234
W00EGS1016047,61075320,149,10262628,98
W00EGS1016181,15813459,191,3565566,183
W00CHZ0103343,105386611,270,6418626,264
W00EGS1016202,11987387,7,6267968,107
W00EGS1016199,109057027,151,2574614,144
W00EGS1016259,18303512,200,2172374,194
W00EGS1016255,26680295,6,20931234,462Data analysisSeedata/counts.ipynbfor turning these into meaningful data. (.ipynbis a jupyter notebook.pip install jupyter && jupyter notebook) brings
up a web browser - you click counts.ipynb to open the script. More work can be gained by understanding the pandas DataFrame api,
but that's for later.Adding testsThese tests exploit parallelism heavily by using producer and consumer queues. The idea is that calculating is
very expensive, but reporting is not. So the runs were written in a poor man's Hadoop by calling the producer action
'map' and the consumer 'reduce'
For most actions, the 'reduce' step only writes output. You don't want to do that in the producing thread
because the output file gets quite large, and each open has to seek to the end.