Package: bdrc-transfer
BDRC Transfer Librarybdrc-transferis a Python library and console script package that
provides SFTP and other services to implement the BDRC workflow to send
BDRC works to a remote site for OCR production, and receive, unpack, and distribute
the resulting works.Copyrighted WorksWhile fair use doctrine allows us to transmit our copies of images of copyrighted works,
there may be an issue with Google Books making them available to their community.Google Books Library Partnership Staff Ben Bunnell described Google Books' copyrght validation process
in an email to BDRC dated 13 Jan 2023:Hi Jim,
We to use the metadata, but the main way is that everything that goes through the Google Books process includes a
copyright verification check as part of the analysis stage. The first few pages of the book are presented to operators
who verify publication dates and location of publication. This info goes through an automated flowchart that
determines
viewability in any given location.For cases where you think the copyright determination is incorrect, you or a general user can open the book on Google
Books, then go to the gear icon (or three-dot menu icon depending on whether you're looking at the new Google Books
interface) /Help/Report problems to request a second review.Best wishes,
BenDebian InstallationOn Debian systems, mysql library is needed:sudo apt install default-libmysqlclient-devInstall audit-tool versionVersion 1.0Beta_2022_05_12or later (audit-tool --versionwill show you the installed
version). Use the latest version fromAudit Tool Releases Pagepip[3] install [--upgrade] [--no-cache-dir] bdrc-transfersome systems only havepip3notpip--upgrade and --no-cache-dirmake sure that the latest release is installed.no-cache-diris usually only
required when testing local disk wheels.--upgradeis for using the pyPi repositoryThen, once only, run:gb-bdrc-initThis copies a google books config from the install directory into the user's.config/gbfolder, making
a backup copy if there is a copy before. The user is responsible for merging their site specific changesGetting StartedManual WorkflowThis is a provisional workflow until all the steps can be automated. Development of automation
for "When the material is ready for conversion" and "Browse to the GB Converted Page" is underway.
TheAutomated Workflowsection of his document will be updated as each release gets this support.The Google Books manual workflow is:Identify works to send to Google BooksCreate and upload the metadata for that list  (upload-metadata)Create a list of paths to the works on that list, and upload that (upload-content)Note that a specially
configured audit-tool validates the content before upload.Wait for some time for Google to process the content. This can be a day to a week.When the material is ready for conversion,GB TBRC InProcess Page- select and save the 'text only'
versionSelect the checkbox for every work (remember there may be multiple pages)click "request conversion" for themWait for some time, and then use GRIN to get the list of files that GB has converted, and which are ready to
download,Browse toGB TBRC Converted Page. For each line you find:In the browser, select the ....pgp.gz file  (they're links) in turn and download it.On the command line:rununpackon the downloaded archiverundistribute_ocron the resulting work structureAutomated WorkflowPreparation and configurationInstall bdrc-transfer >= 0.0.4. v 0.0.4 implements the automated "conversion request step" (below)Choose a user to host acrontabentry. The user's environment must contain the environment variables listed in **
Runtime** below. The recommended way is to use the user's interactivebashenvironment, as shown here. Be sure
that the file referenced in BASH_ENV passes control to some script which initializes all the variables. (Typically,
.bashrc, but probably some variant of it)# m h  dom mon dow   command# * *   *   *   *     BASH_ENV=~/.profile request-conversion-serviceSchedule the crontab entry shown aboveWorkflowThe Google Books automated workflow is:Identify works to send to Google BooksCreate and upload the metadata for that list  (upload-metadata)Create a list of paths to the works on that list, and upload that (upload-content)Note that a specially
configured audit-tool validates the content before upload.The crontab entryrequest-conversion-service(see above) will poll the Google Books server and look for volumes
available for conversion, and will request them.The crontab entryprocess-converted(inbdrc-transfer 0.0.5) will:Poll the Google Books server for volumes which are ready to download.Download, unpack, and distribute the OCR'd volume and support.Backlog processingThere are some utilities that can help in setting up the process
For example, we have manually downloaded and unpacked items before.
To trigger a re-distribution, we can signal again that they've been downloaded. The command line toolmark-downloaded [-i [ paths | - ] path, .....marks in the internal tracking system that those items have been
downloaded. The items
must have the file name format{parent_path}/WorkRid-ImageGroupRid.tar.gzRuntimeEnvironment configurationbdrc-transferrequires these environment variables, unless overridden on the command line.
(Overriding is not recommended in production)GRIN_CONFIG- Path to the configuration file, which contains authorization and other essential data.
The name and contents of this file should be closely held in the development team.Environment variables which
v<= 0.0.4 read are now in this file.LoggingOne requirement of this package is that there be a single, authoritative log of activities. In development,
there will be testing attempts. It should be easy to add the logs of these testing attempts to the single log.
Eachgb_ocroperation defines a tuple ofactivityanddestination.Theactivityvalues are:uploadrequest_conversionunpackdistributeand thedestinationvalues are:metadatacontentThe resulting set of log files this package creates are:upload_metadata-activity.logupload_metadata-runtime.logupload_content-activity.logupload_content-runtime.logrequest_conversion-activity.logrequest_conversion-runtime.logtransfer-activity.logtransfer-runtime.logunpack-activity.logunpack-runtime.logRuntime logThis is a free-form console log for diagnostic and informational purposes. Runtime logs'
filenames end with-runtime.logContent logThis is the canonical log file for the activity. Each activity module in thegb_ocrIts structure is optimized for
programmatic import, not human readability. These log files end in-content.log(the older ones end
in-activity.logand may have a different format)** v 0.1.12 update ** The canonical log has been moved into a database. The database is accessed through theAORunActivityLog.db_activity_loggerclass. However, the log file can still
be used to update the database with activity from the canonical log,
in case of##### Log file naming
Log files are intended to be continuous, and are not concurrency safe. *Activity logs* are intended to be singular
across the whole BDRC network, so there *must* be only one activity instance writing at a time.
(As of 7 Jun 2022, this is not enforced)Available commandsunpack
               relocate-downloads
               gb-convert
               move-downloads
               upload-metadata
               distribute-ocr
               upload-content
                request-conversion (request-conversion-service)
                request-download (request-download-service)Common OptionsAll commands in this section share these common options:optionalarguments:-h,--helpshowthishelpmessageandexit-lLOG_HOME,--log_homeLOG_HOMEWherelogsarestored-seemanual-n,--dry_runConnectonly.Donotupload-d{info,warning,error,debug,critical},--debug_level{info,warning,error,debug,critical}choicevaluesarefrompythonloggingmodule-z,--log_after_fact(expostfacto)logasuccessfulactivityafteritwasperformedoutofband-i[FILE],--input_file[FILE]filestoread.use-forstdinupload-metadatausage:upload-metadata[-h][-lLOG_HOME][-n][-d{info,warning,error,debug,critical}][-z][-i[FILE]][work_rid]Createsandsendsmetadatatogb

positionalarguments:work_ridWorkIDupload-content❯upload-content--help
usage:upload-content[-h][-lLOG_HOME][-n][-d{info,warning,error,debug,critical}][-z][-i[FILE]][-g][work_path]uploadstheimagesinaworktoGB.Canuploadallorsomeimagegroups(see--image_groupoption)...commonarguments-g,--image_groupTrueifpathsaretoimagegroupunpackusage:unpack[-h][-lLOG_HOME][-n][-d{info,warning,error,debug,critical}][-z][-i[FILE]][src]Unpacksanartifact

positionalarguments:srcxxx.tar.gz.gpgfiletounpackUnpacks a downloaded GB processed artifact (Note that the download is not FTP,
so there is no API to download. In 0.0.1, this is a manual operation)See the section Distribution format for the output documentationgb-convertThis is a stub function, which simulates requesting a conversion from the Google books
web UI. It simply logs the fact that the user has checked a whole list of items to convert.
Usually the user will have to download the list from gb, extract the image group rids, and feed them
into this program.usage:gb-convert[-h][-lLOG_HOME][-n][-d{info,warning,error,debug,critical}][-z][-i[FILE]][image_group]Requestsconversionofanuploadedcontentimagegroup

positionalarguments:image_groupworkRid-ImageGroupRid-nofilesuffixesftp-transferThis is a low level utility function, which should not generally be used in the workflow.usage:ftp-transfer[-h][-lLOG_HOME][-n][-d{info,warning,error,debug,critical}][-z][-i[FILE]][-m|-c][-p|-g]src[dest]Uploadsafiletoaspecificpartnerserver,definedbyasectionintheconfigfile

positionalarguments:srcsourcefilefortransferdest[Optional]destinationfile-defaultstobasenameofsourceoptionalarguments:filestoread.use-forstdin-m,--metadataActonmetadatatarget-c,--contentActonthecontenttarget-p,--putsendto-g,--getgetfrom(NOTIMPLEMENTED)LaunchingDefine the environment variableGB_CONFIGto point to the configuration file for the project. The configuration file
is the access point to GB's sftp host, and is tightly controlled.Activity Tracking and LoggingActivity tracing is the responsibility of thelog_ocrpackage.
Thelog_ocrhas a public moduleAORunLog.pywhich contains theAORunActivityLogclass. This class offers three
interfaces to its clients. These are separated into two groups:loggingimplementations, and database implementationsLoggingThese are Pythonlogginginstances, and offer the completelogginginterfaceactivity_loggerruntime_loggerDatabase implementationThe database implementation is a replacement for the activity logger, which is a simple canonical journal of GB OCR
processing.activity_db_loggerThis is an instance of classlog_ocr.GbOcrTrack.GbOcrTracker. This exposes the following
methods:add_content_request - Records a content process step:uploadrequest_conversiondownload image groups which GB has processeddistributeget_ready_to_convert: Gets a list of image groups which GB has received, but we have not requested conversionget_converted: Gets a list of image groups which GB has converted, but we have not downloaded.The propertylog_ocr.AORunLog.activity_db_loggeris the replacement for the "activity" tracking log discussed below.
It does not use the pythonloggingAPI, but its own specific methods, which are found in `log_ocr.LoggingLog storeThe default directory for logging can be given in these directives:the current working directory is the default, in the absence of these next entries.Environment variableRUN_ACTIVITY_LOG_HOME.the-l/--log_homeargument toftp-transfer. Overrides the environment variable if givenLog filesftp_transferlogs two kids of activity:runtime logs,transfer-runtime.logdescribing details of an operation. The content of this log is affected by
the-dflag.activity logs.transfer-activity.log. They provide limited, but auditable information on:the activity subject (metadata or content)the activity outcome (success or fail)
It is the caller's responsibility to aggregate activity logs into a coherent view of activity.Log formatRuntime Formatshort date time:message:content descriptorExample:06-03 15:29:INFO:upload success /Users/jimk/dev/tmp/aog1/META/marc-W2PD17457.xml:metadataActivity FormatDate mm-DD-YYYY HH-MM-SS:operation:status:message:content descriptorExample:06-06-2022 20-28-06:get:error:/Users/jimk/dev/tmp/aog1/META/marc-W2PD17457.xml:metadata:Distribution FormatThis section defines the format of the OCR distribution on BDRC's OCR servers. It is the final result of the
discussions in Github buda-base archive-ops-694 (no URL given, private repository)The distribution format for a typical work, and one image group in that work, is
shown here:❯ tree --filesfirst  Works
Works/
└── a9/
    └── W1PD12345/
        └── google_books/
            └── batch_2022/
                ├── info.json
                ├── info/
                │   ├── W1PD12345-I1PD12345/
                │   │      ├── gb-bdrc-map.json
                │   │      └── TBRC_W1PD12345-I1PD12345.xml
                │   └── W1PD12345-I1PD12..../               
                └── output/
                    ├── W1PD12345-I1PD12345/
                    │    ├── html.zip
                    │    ├── images.zip
                    │    └── txt.zip
                    └── W1PD12345-I1PD12..../Folder structureWork level foldersWorks/{hash}/{wid}/{ocrmethod}/{batchid}/Where:where:{hash}is the well-known hash (2 first hexa digits of the md5 of the W id){wid}is also well-known (ex:W22084){ocrmethod}should bevision/for Google OCR{batchid}should be a unique batch id, it doesn't need to be in alphabetical order, it just needs to be unique per
wid+ocrmethod (in the Google Books delivery, this is the literal 'batch_2022'){batchid}contains one file and two folders:info.jsoninfooutputIn the following discussion,{wid}-{iid}refers to the WorkRID-ImageGroupRID tuple as a string
(W1PD12345-I1PD12345, in this example)info.json{wid}/info.jsoncontains:{
  "timestamp" : "X",
  "html": "html.zip",
  "txt": "txt.zip",
  "images": "images.zip"
}It is uploaded with every image group, so timestamp will always be the latest upload, even if all the image groups are
not present in OCR yet.
However, because our image group processing is independent, there's no flag to say when all the image groups in a run
are done (there's not even a notion of a run - buda-base/ao-google-books#23 requests that implementation.The keyshtmltxtandimagesare finding aids - they reference the filenames
underoutput/{wid}-{iid}(Note this forces every image group under the Work to be in this structure)info/This is a dictionary of metadata. It contains, for each{wid}-{iid}that has been processed,gb-bdrc-map.json: mapping between BUDA image list and OCR derived image list. The BDRC Google Books process
creates this artifact.TBRC_{wid}-{iid}.xml: The Google Books creation process delivers this file, which BDRC Google Books process
relocates from the original position here. This file contains PREMIS metadata for the image group.output/Output contains only folders for each{wid}-{iid}/in the work
Each of these contains only three files, each of which is an archive of
Google Books generated content.html.zip- HOCR files (OCR content in HTML format)images.zip- Generated images from which Google Books derived the OCRtxt.zip- Unicode text that Google Books generatedChangelogVersionChanges0.1.20833ba79Retry 404sec7812cUpload in small bites0.1.19419daebAdd missing lib reference0.1.18e0e4adcUse better image detector0.1.176f9a5b88console logging of headerb110073Staging for get custom query603d0caMove ORM to bdrc-db-lib0.1.16d842f98Segment request conversion requests0.1.15Database object refactoring0.1.85a6b000Uploadstandalone image groups