Package: s3-ocr
s3-ocrTools for running OCR against files stored in S3Background on this project:s3-ocr: Extract text from PDF files stored in an S3 bucketInstallationInstall this tool usingpip:pip install s3-ocrDemoYou can see the results of running this tool against three PDFs from the Internet Archive (one,two,three) inthis example tablehosted usingDatasette.Starting OCR against PDFs in a bucketThestartcommand takes a list of keys and submits them toTextractfor OCR processing.You need to have AWS configured using environment variables, credentials file in your home directory or a JSON or INI file generated usings3-credentials.You can start the process running like this:s3-ocr start name-of-your-bucket my-pdf-file.pdfThe paths you specify should be paths within the bucket. If you stored your PDF files in folders inside the bucket it should look like this:s3-ocr start name-of-your-bucket path/to/one.pdf path/to/two.pdfOCR can take some time. The results of the OCR will be stored intextract-outputin your bucket.To process every file in the bucket with a.pdfextension use--all:s3-ocr start name-of-bucket --allTo process every file with a.pdfextension within a specific folder, use--prefix:s3-ocr start name-of-bucket --prefix path/to/folders3-ocr start --helpUsage: s3-ocr start [OPTIONS] BUCKET [KEYS]...

  Start OCR tasks for PDF files in an S3 bucket

      s3-ocr start name-of-bucket path/to/one.pdf path/to/two.pdf

  To process every file with a .pdf extension:

      s3-ocr start name-of-bucket --all

  To process every .pdf in the PUBLIC/ folder:

      s3-ocr start name-of-bucket --prefix PUBLIC/

Options:
  --all                 Process all PDF files in the bucket
  --prefix TEXT         Process all PDF files within this prefix
  --dry-run             Show what this would do, but don't actually do it
  --no-retry            Don't retry failed requests
  --access-key TEXT     AWS access key ID
  --secret-key TEXT     AWS secret access key
  --session-token TEXT  AWS session token
  --endpoint-url TEXT   Custom endpoint URL
  -a, --auth FILENAME   Path to JSON/INI file containing credentials
  --help                Show this message and exit.Checking statusThes3-ocr status <bucket-name>command shows a rough indication of progress through the tasks:% s3-ocr status sfms-history
153 complete out of 532 jobsIt compares the jobs that have been submitted, based on.s3-ocr.jsonfiles, to the jobs that have their results written to thetextract-output/folder.s3-ocr status --helpUsage: s3-ocr status [OPTIONS] BUCKET

  Show status of OCR jobs for a bucket

Options:
  --access-key ...Inspecting a jobThes3-ocr inspect-job <job_id>command can be used to check the status of a specific job ID:% s3-ocr inspect-job b267282745685226339b7e0d4366c4ff6887b7e293ed4b304dc8bb8b991c7864
{
  "DocumentMetadata": {
    "Pages": 583
  },
  "JobStatus": "SUCCEEDED",
  "DetectDocumentTextModelVersion": "1.0"
}s3-ocr inspect-job --helpUsage: s3-ocr inspect-job [OPTIONS] JOB_ID

  Show the current status of an OCR job

      s3-ocr inspect-job <job_id>

Options:
  --access-key ...Fetching the resultsOnce an OCR job has completed you can download the resulting JSON using thefetchcommand:s3-ocr fetch name-of-bucket path/to/file.pdfThis will save files in the current directory with names like this:4d9b5cf580e761fdb16fd24edce14737ebc562972526ef6617554adfa11d6038-1.json4d9b5cf580e761fdb16fd24edce14737ebc562972526ef6617554adfa11d6038-2.jsonThe number of files will vary depending on the length of the document.If you don't want separate files you can combine them together using the-c/--combineoption:s3-ocr fetch name-of-bucket path/to/file.pdf --combine output.jsonTheoutput.jsonfile will then contain data that looks something like this:{
  "Blocks": [
    {
      "BlockType": "PAGE",
      "Geometry": {...}
      "Page": 1,
      ...
    },
    {
      "BlockType": "LINE",
      "Page": 1,
      ...
      "Text": "Barry",
    },s3-ocr fetch --helpUsage: s3-ocr fetch [OPTIONS] BUCKET KEY

  Fetch the OCR results for a specified file

      s3-ocr fetch name-of-bucket path/to/key.pdf

  This will save files in the current directory called things like

      a806e67e504fc15f...48314e-1.json     a806e67e504fc15f...48314e-2.json

  To combine these together into a single JSON file with a specified name, use:

      s3-ocr fetch name-of-bucket path/to/key.pdf --combine output.json

  Use "--output -" to print the combined JSON to standard output instead.

Options:
  -c, --combine FILENAME  Write combined JSON to file
  --access-key ...Fetching just the text of a pageIf you don't want to deal with the JSON directly, you can use thetextcommand to retrieve just the text extracted from a PDF:s3-ocr text name-of-bucket path/to/file.pdfThis will output plain text to standard output.To save that to a file, use this:s3-ocr text name-of-bucket path/to/file.pdf > text.txtSeparate pages will be separated by three newlines. To separate them using a----horizontal divider instead add--divider:s3-ocr text name-of-bucket path/to/file.pdf --dividers3-ocr text --helpUsage: s3-ocr text [OPTIONS] BUCKET KEY

  Retrieve the text from an OCRd PDF file

      s3-ocr text name-of-bucket path/to/key.pdf

Options:
  --divider             Add ---- between pages
  --access-key ...Avoiding processing duplicatesIf you move files around within your S3 buckets3-ocrcan lose track of which files have already been processed. This can lead to additional Textract charges for processing should you runs3-ocr startagainst those new files.Thes3-ocr dedupecommand addresses this by scanning your bucket for files that have a new name but have previously been processed. It does this by looking at theETagfor each file, which represents the MD5 hash of the file contents.The command will write out new.s3ocr.jsonfiles for each detected duplicate. This will avoid those duplicates being run those duplicates through OCR a second time should yo runs3-ocr start.s3-ocr dedupe name-of-bucketAdd--dry-runfor a preview of the changes that will be made to your bucket.s3-ocr dedupe --helpUsage: s3-ocr dedupe [OPTIONS] BUCKET

  Scan every file in the bucket checking for duplicates - files that have not
  yet been OCRd but that have the same contents (based on ETag) as a file that
  HAS been OCRd.

      s3-ocr dedupe name-of-bucket

Options:
  --dry-run             Show output without writing anything to S3
  --access-key ...Changes made to your bucketTo keep track of which files have been submitted for processing,s3-ocrwill create a JSON file for every file that it adds to the OCR queue.This file will be called:path-to-file/name-of-file.pdf.s3-ocr.jsonEach of these JSON files contains data that looks like this:{"job_id":"a34eb4e8dc7e70aa9668f7272aa403e85997364199a654422340bc5ada43affe","etag":"\"b0c77472e15500347ebf46032a454e8e\""}The recordedjob_idcan be used later to associate the file with the results of the OCR task intextract-output/.Theetagis the ETag of the S3 object at the time it was submitted. This can be used later to determine if a file has changed since it last had OCR run against it.This design for the tool, with the.s3-ocr.jsonfiles tracking jobs that have been submitted, means that it is safe to runs3-ocr startagainst the same bucket multiple times without the risk of starting duplicate OCR jobs.Creating a SQLite index of your OCR resultsThes3-ocr index <bucket> <database_file>command creates a SQLite database containing the results of the OCR, and configures SQLite full-text search against the text:% s3-ocr index sfms-history index.db
Fetching job details  [####################################]  100%
Populating pages table  [####################----------------]   55%  00:03:18The schema of the resulting database looks like this (excluding the FTS tables):CREATETABLE[pages]([path]TEXT,[page]INTEGER,[folder]TEXT,[text]TEXT,PRIMARYKEY([path],[page]));CREATETABLE[ocr_jobs]([key]TEXTPRIMARYKEY,[job_id]TEXT,[etag]TEXT,[s3_ocr_etag]TEXT);CREATETABLE[fetched_jobs]([job_id]TEXTPRIMARYKEY);The database is designed to be used withDatasette.s3-ocr index --helpUsage: s3-ocr index [OPTIONS] BUCKET DATABASE

  Create a SQLite database with OCR results for files in a bucket

Options:
  --access-key ...DevelopmentTo contribute to this tool, first checkout the code. Then create a new virtual environment:cd s3-ocr
python -m venv venv
source venv/bin/activateNow install the dependencies and test dependencies:pip install -e '.[test]'To run the tests:pytestTo regenerate the README file with the latest--help:cog -r README.md