Boriscii commited on
Commit
256ca36
1 Parent(s): 84075cc

Update README.md

Browse files

# LayoutLM-Byne
## The new SOTA in page retrieval from visually-rich documents.

[![Logo](https://armalytix.s3.eu-west-2.amazonaws.com/TRUST+THE+COUNSEL+(1).png "Logo")](https://bynedocs.com "Logo")

We're glad to introduce one of the first document page embedding models, LayoutLM-Byne.

With the rise of multimodal LLMs, there is a growing adoption of applying models directly to a document without pre-processing it first, as was done before with RAG. This approach is significantly more robust than text-only RAG on a large subset of documents, especially visually rich ones.

On the other hand, there is a significant lack of research focused on extracting a relevant page from a PDF or a DOCX document. Most practitioners would parse the page into text and apply regular text embeddings to the text, losing much positional context in the process.

LayoutLM [1] is an excellent solution for the problems because, at its core, it is a regular BERT-alike model, but it is uniquely capable of embedding positional information about the text alongside the text itself.

We have fine-tuned the model on the DocVQA [2] dataset, far surpassing the current SOTA (all-mpnet-base-v2) [3]:

| Model | HR@3 | HR@5 | HR@10 |
|-------|------|------|-------|
| all-mpnet-base-v2 (Baseline) | 0.2505 | 0.2941 | 0.3624 |
| LayoutLM (Our Model) | 0.3159 | 0.3909 | 0.5019 |
| Relative Improvement | +26.1% | +32.9% | +38.5% |

### Usage
Please refer to the Colab workbook or the blog post to learn more!

### Get in touch
Reach out to [borys.nadykto@bynesoft.com](mailto:borys.nadykto@bynesoft.com) if you'd like help with deploying the model in commerical setting.

Files changed (1) hide show
  1. README.md +8 -3
README.md CHANGED
@@ -1,3 +1,8 @@
1
- ---
2
- license: mit
3
- ---
 
 
 
 
 
 
1
+ ---
2
+ license: mit
3
+ datasets:
4
+ - lmms-lab/DocVQA
5
+ language:
6
+ - en
7
+ pipeline_tag: sentence-similarity
8
+ ---