Package: vila
VILA🌴IncorporatingVIsualLAyout Structures for Scientific Text ClassificationVILA🌴IncorporatingVIsualLAyout Structures for Scientific Text ClassificationMotivationInstallationUsageAccessing The S2-VLUE DatasetsRun Inference/PredictionModel WeightsMMDA VILA ExamplePDFPredictors in VILATrainingDirectory StructureObtaining the DatasetTraining ScriptsEvaluation ToolkitNoteCitationMotivationScientific papers typically organize contents in visual groups like text blocks or lines, and text within each group usually have the same semantics. We explore different approaches for injecting the group structure into the text classifiers, and build models that improves the accuracy or efficiency of the scientific text classification task.InstallationAfter cloning the github repo, you can either install thevilalibrary or just install the dependencies:gitclonegit@github.com:allenai/VILA.gitcdVILAcondacreate-nvilapython=3.6
pipinstall-e.# Install the `vila` librarypipinstall-rrequirements.txt# Only install the dependenciesWe tested the code and trained the models usingPython≥3.6,PyTorch==1.7.1, andtransformers==4.4.2.UsageAccessing The S2-VLUE DatasetsPlease check the detailed instructions indatasets/README.md.Run Inference/PredictionModel WeightsWe've uploaded a collection of pre-trained models to HuggingFace's model Hub:Weights NameModelDatasetF1Latencylayoutlm-base-uncasedbaselinedocbank91.0652.56allenai/ivila-block-layoutlm-finetuned-docbankiviladocbank92.00-allenai/hvila-block-layoutlm-finetuned-docbankhviladocbank87.7816.37allenai/hvila-row-layoutlm-finetuned-docbankhviladocbank91.2728.07layoutlm-base-uncasedbaselinegrotoap292.3452.56allenai/ivila-block-layoutlm-finetuned-grotoap2ivilagrotoap293.38-allenai/hvila-block-layoutlm-finetuned-grotoap2hvilagrotoap292.3716.37allenai/hvila-row-layoutlm-finetuned-grotoap2hvilagrotoap291.6528.07MMDA VILA ExampleMMDAis our newly designed toolkit that provides flexible supports for PDF document analysis. Please check the VILA predictor exampleherefor more details.PDFPredictors in VILAIn VILA repo, we also implemented a set ofPDFPredictors. Please refer to the example code below:importlayoutparseraslp# For visualizationfromvila.pdftools.pdf_extractorimportPDFExtractorfromvila.predictorsimportHierarchicalPDFPredictor# Choose from SimplePDFPredictor,# LayoutIndicatorPDFPredictor,# and HierarchicalPDFPredictorpdf_extractor=PDFExtractor("pdfplumber")page_tokens,page_images=pdf_extractor.load_tokens_and_image(f"path-to-your.pdf")vision_model=lp.EfficientDetLayoutModel("lp://PubLayNet")pdf_predictor=HierarchicalPDFPredictor.from_pretrained("allenai/hvila-block-layoutlm-finetuned-docbank")foridx,page_tokeninenumerate(page_tokens):blocks=vision_model.detect(page_images[idx])page_token.annotate(blocks=blocks)pdf_data=page_token.to_pagedata().to_dict()predicted_tokens=pdf_predictor.predict(pdf_data,page_token.page_size)lp.draw_box(page_images[idx],predicted_tokens,box_width=0,box_alpha=0.25)TrainingDirectory StructureVILA
├─ checkpoints  # For all trained weights 
│  └─ grotoap2  # For each dataset                                 
│     ├─ baseline  # For the experiment type, e.g., baseline, ivila, hvila, ...
│     │  └─ bert-base-uncased  # For the used base model, e.g., bert-base-uncased. 
│     │     ├─ checkpoint-199999                                
│     │     ├─ checkpoint-299999                                 
│     │     ├─ all_results.json                                       
│     │     └─ pytorch_model.bin                         
│     └─ ivila-BLK-row                           
│        └─ microsoft-layoutlm-base-uncased 
└─ data                                       
   ├─ docbank
   ├─ grotoap2
   └─ s2-vlObtaining the DatasetWe provide the scripts to help you easily download and save the datasets into the aforementioned structure. Please follow the instructions indatasets/README.md.Training ScriptsAll training scripts are in the./scriptsfolder.Train the baseline modelscdscripts# bash train_baseline.sh [dataset-name] [base-model-name]bashtrain_baseline.shgrotoap2bert-base-uncased
bashtrain_baseline.shdocbankmicrosoft/layoutlm-base-uncasedTrain the I-VILA modelscdscripts# bash train_ivila.sh [dataset-name] [how-to-obtain-layout-indicators] [used-special-token] [base-model-name]bashtrain_ivila.shgrotoap2rowBLKmicrosoft/layoutlm-base-uncased# Row is an alias for textlinebashtrain_ivila.shdocbankblockSEPbert-base-uncased# We can also use the default special tokens like SEPbashtrain_ivila.shs2-vlsentenceBLKroberta-base# We can also extract the sentence breaks using spacy and use them as indicators.Train the H-VILA modelscdtools
pythoncreate_hvila_model_base_weights.pycd../scripts# bash train_hvila.sh \#  [dataset-name] \#  [H-VILA-names] \#  [Group-Encoder-Output-Aggregation-Function] \#  [How-to-Obtain-Bounding-Box] \#  [Use-textline-or-block-as-the-group]bashtrain_hvila.sh\grotoap2\weak-strong-layoutlm\average\first\rowIn the above example, we use the:averageof the group encoder outputs for all tokens as the group representationthe bounding box of thefirsttoken as the group's bounding boxtextline (or row) to construct the groupsEvaluation ToolkitThe evaluation toolkit can generate a detailed report for the prediction accuracy (marco F1 scores) and Visual Layout consistency (group entropy) for the prediction filestest_predictions.csvproduced by the training scripts.Generate reports for a group of experiments for a specific datasetcdtoolspythongenerate-eval.py--dataset_namegrotoap2--experiment_namebaseline# It will create a _reports folder in ../checkpoints/grotoap2/baseline and store the# scores in the report.csv file.Generate reports for all experiments for a specific datasetcdtoolspythongenerate-eval.py--dataset_namegrotoap2# It will create reports for all experiments in the ../checkpoints/grotoap2/ folder# Also it will aggregate all the results and save them in ../checkpoints/grotoap2/_reportsGenerate reports for per-class accuracycdtoolspythongenerate-eval.py--dataset_namegrotoap2--experiment_namebaseline--store_per_class# In additiona to the report.csv file, it will also generate a report_per_class.csv# table in the corresponding folder.Note: this evaluation toolkits might take a long time to run as calculing the group entropy may take long.NoteIn order to support the AutoModel API, we changed the default transformers requirements to>=4.5instead of4.4.2. If you're working on reproducing the results, you might want to downgrade the transformers version to 4.4.2 to get the matching results.Citation@article{shen2022vila,
  title={VILA: Improving structured content extraction from scientific PDFs using visual layout groups},
  author={Shen, Zejiang and Lo, Kyle and Wang, Lucy Lu and Kuehl, Bailey and Weld, Daniel S and Downey, Doug},
  journal={Transactions of the Association for Computational Linguistics},
  volume={10},
  pages={376--392},
  year={2022},
  publisher={MIT Press}
}