Package: quipucamayoc
Quipucamayoc: tools for digitizing historical dataquipucamayocis a Python package that simplifies the extraction of historical data from scanned images and PDFs.
It's designed to be modular and so it can be used together with other existing tools, and can be extended easily by users.For an overview of how to usequipucamayocto digitize historical data, seethis research article, which amongst other things details the different steps involved, the methods used, and provides practical examples.
For an user guide, documentation, and installation instructions, seehttp://scorreia.com/software/quipucamayoc/(TODO).If you want to contribute by improving the code or extending its functionality (much welcome!), headhere.InstallationPipTo manage quipucamayoc using pip, open the command line and run:pip install quipucamayocto installpip install "quipucamayoc[dev]"to include extra dependencies used when developing the codepip install -U quipucamayocto upgradepip uninstall quipucamayocto removeNote thatquipucamayochas been tested against Python 3.10 and newer versions, but should also work with Python 3.9.Git InstallAfter cloning the repo to your computer and navigating to the quipucamayoc folder, run:pip install .to install the package locallypip install -e .to install locally with a symlink so changes are automatically updated (recommended for developers)After installationAWSAWS configuration is quite cumbersome, so it has been automated. To do so, follow these four steps:Downloadand install theawscommand line interface (CLI).Update:quipucamayocinstalls theawsclipackage so this step might not be necessary anymore.Configureyour credentials withaws configure. This requires an Amazon/AWS account.From the command line, run the quipucamayoc commandquipu aws installNotes:You can avoid step 1 by directly [writing your credentials[(https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html)] to thecredentialsfile.Steps 3-4 are also available from within Python in thesetup_textract()andtest_textract()functions.If you want to remove all quipucamayoc artifacts from your AWS account, you can runquipu aws uninstallfrom the command line.The defaultAWS regionisaws-east-1. To use other regions, use the--region <name>option.UsageFrom the command line, you can extract tables using AWS viaquipu extract-tables --filename <myfile.pdf>TODOAutomatically set up Textract pipelineExpose key functions as command line toolsAllow parallel (async?) tasks. Useful for OpenCV (CPU-intensive) and Textract calls (IO-intensive). Consider alsouvloopInclude Poppler by default on WindowsAdd mypy/(flake8|black)ContributingFeel free to submit push requests. For consistency, code should comply withpep8(as long as its reasonable), and with the style guides by@kennethreitzandgoogle. Read morehere.Citation(Download BibTex file here)As textSergio Correia, Stephan Luck: “Digitizing Historical Balance Sheet Data: A Practitioner's Guide”, 2022;arXiv:2204.00052.As BibTex@misc{quipucamayoc,Author={Correia, Sergio and Luck, Stephan},Title={Digitizing Historical Balance Sheet Data: A Practitioner's Guide},Year={2022},eprint={arXiv:2204.00052},journal={arXiv preprint arXiv:2204.00052}}AcknowledgmentsQuipucamayoc is built upon the work and improvements of many users and developers, from which it was heavily inspired, such as:pdftabextractIt is also relies for most of its work on the following open source projects:PythonNumPyOpenCVandOpenCV-PythonPopplerLicenseQuipucamayoc is developed under theGNU Affero GPL v3 license.Why "quipucamayoc"?Thequipucamayocswere the Inca empire officials in charge of desciphering (amonst other things) accounting information stored in quipus. Our goal for this package is to act as a sort of quipucamayoc, helping researchers in desciphering and extracting historical information, particularly balance sheets and numerical records.