Package: appjsonify
appjsonify: An Academic Paper PDF-to-JSON Conversion Toolkitappjsonify[^1] is a handy PDF-to-JSON conversion tool for academic papers implemented in Python.appjsonifyallows you to obtain a structured JSON file that can be easily used for various downstream tasks such as paper recommendation, information extraction, and information retrieval from papers.[^1]:AcademicPaperPDFjsonifyRequirementsLinux or macOS (Not tested on Windows)Python 3.10 or laterpdfplumberregistrabletqdmpillowpdf2imagetorchdetectron2Please manually install it based onthe instructions.InstallationPrerequisitesIf your environment does not havepoppler, please install it.
This is necessary to obtain PDF images usingpdf2image.
For more details, refer toPrerequisites.Released versionpipinstallappjsonify
python-mpipinstall'git+https://github.com/facebookresearch/detectron2.git'Editable version (Beta)gitclonehttps://github.com/hitachi-nlp/appjsonify.git
python-mpipinstall--editable.
python-mpipinstall'git+https://github.com/facebookresearch/detectron2.git'Usageappjsonifyoffers two options to structure your paper PDF file into a JSON file.Use the existing templatesSuitable if a paper adopts theAAAI,ACL,ICML,ICLR,NeurIPS,IEEE,ACM, orSpringerstyles. SeeTemplatesfor more details.Configure pipelines and parameters by yourselfIf a paper does not adopts the above formats, you need to specify the processing pipeline and its parameters. Please refer toBuild your own pipelinefor further information.Templatesappjsonifyprovides two types of the templates for each of the following paper types:AAAI,ACL,ICML,ICLR,NeurIPS,IEEE,ACM, andSpringer.
One is more accurate but slower due to the use of machine learning based models, the other is less accurate but faster due to its rule based approach.AAAI papersBetter performance but slowerappjsonify/path/to/pdf/dir/or/path/path/to/output/dir--paper_typeAAAIIf your environment has a GPU(s), it is better to also specify--detectron_device_mode cudato speed up the process.Faster but a bit noisyappjsonify/path/to/pdf/dir/or/path/path/to/output/dir--paper_typeAAAI2ACL papersBetter performance but slowerappjsonify/path/to/pdf/dir/or/path/path/to/output/dir--paper_typeACLFaster but a bit noisyappjsonify/path/to/pdf/dir/or/path/path/to/output/dir--paper_typeACL2ICML papersBetter performance but slowerappjsonify/path/to/pdf/dir/or/path/path/to/output/dir--paper_typeICMLFaster but a bit noisyappjsonify/path/to/pdf/dir/or/path/path/to/output/dir--paper_typeICML2ICLR papersBetter performance but slowerappjsonify/path/to/pdf/dir/or/path/path/to/output/dir--paper_typeICLRFaster but a bit noisyappjsonify/path/to/pdf/dir/or/path/path/to/output/dir--paper_typeICLR2NeurIPS papersBetter performance but slowerappjsonify/path/to/pdf/dir/or/path/path/to/output/dir--paper_typeNeurIPSFaster but a bit noisyappjsonify/path/to/pdf/dir/or/path/path/to/output/dir--paper_typeNeurIPS2IEEE papersCurrently only tested with IEEE BigData papers.Better performance but slowerappjsonify/path/to/pdf/dir/or/path/path/to/output/dir--paper_typeIEEEFaster but a bit noisyappjsonify/path/to/pdf/dir/or/path/path/to/output/dir--paper_typeIEEE2ACM papersCurrently only tested with TALLIP papers.Better performance but slowerappjsonify/path/to/pdf/dir/or/path/path/to/output/dir--paper_typeACMFaster but a bit noisyappjsonify/path/to/pdf/dir/or/path/path/to/output/dir--paper_typeACM2Springer papersBetter performance but slowerappjsonify/path/to/pdf/dir/or/path/path/to/output/dir--paper_typeSpringerFaster but a bit noisyappjsonify/path/to/pdf/dir/or/path/path/to/output/dir--paper_typeSpringer2Useful parameters--verbose: If you want to check the intermediate processing results, please set this flag. The log files will be saved underoutput_dir. Optionally, you can use the following four flags to add the corresponding information.--show_pos: Bounding box information.--show_font: Font name and size information.--show_style: Style information (e.g.,section,body,abstract, etc.)--show_meta: Supplementary information (e.g., information on objects and footnotes.)--insert_page_break: Insert breaks between pages.--save_image: If you are using a more accurate but slower version of templates orload_objects_with_ml,appjsonifycan save detected table and figure images if this flag is set. In addition to this, please also specify the output directory path as--output_image_dir.Build your own pipelineappjsonifyalso allows users to build their own academic paper PDF-to-JSON processing pipeline.
For more details, please refer toAvailable ModulesandDocument Handling inappjsonify.How to add your own moduleUsers can add their own modules toappjsonifyfor more flexible document processing.
To add modules,appjsonifymust be installed ineditablemode.
SeeCustomizeappjsonifyfor more details and feel free to make a PR if you wish to add your module to this repository and package!Contributing and Future WorkContributions are more than welcome! Feel free to raise an issue and/or make a PR.
Possible future work is as follows:Better documentationMore paper templatesMore robust references extractionPowerful mathematical equation supportRobust algorithm description detectionMultilingual supportAdd more test scriptsCitationIf you useappjsonifyin your work, please cite the following.@article{Yamaguchi2023appjsonify,
  title={appjsonify: An Academic Paper PDF-to-JSON Conversion Toolkit},
  author={Atsuki Yamaguchi and Terufumi Morishita},
  year={2023}
}License© 2023 Atsuki Yamaguchi and Terufumi Morishita (Hitachi, Ltd.)This work is licensed under the MIT license unless specified.appjsonifyuses the follwoing publicly available works.pdfplumberby Jeremy Singer-Vine (MIT License).registrableby epwalsh (Apache License 2.0).tqdm(MIT License, Mozilla Public License 2.0 (MPL 2.0)).pillowby Jeffrey A. Clark (Historical Permission Notice and Disclaimer License).pdf2imageby Edouard Belval (MIT License).torch(BSD-style license).Detectron2by Facebook AI Research (Apache License 2.0) indetectron2_demo.DocBankpretrained model by Microsoft Research Asia (Apache License 2.0) indocbank.py.TableBankpretrained model by Microsoft Research Asia (Apache License 2.0) intablebank.py.PubLayNetpretrained model by hpanwar08 (Apache License 2.0) inpublaynet.py.