Update README.md
Browse files# LayoutLM-Byne
## The new SOTA in page retrieval from visually-rich documents.
[.png "Logo")](https://bynedocs.com "Logo")
We're glad to introduce one of the first document page embedding models, LayoutLM-Byne.
With the rise of multimodal LLMs, there is a growing adoption of applying models directly to a document without pre-processing it first, as was done before with RAG. This approach is significantly more robust than text-only RAG on a large subset of documents, especially visually rich ones.
On the other hand, there is a significant lack of research focused on extracting a relevant page from a PDF or a DOCX document. Most practitioners would parse the page into text and apply regular text embeddings to the text, losing much positional context in the process.
LayoutLM [1] is an excellent solution for the problems because, at its core, it is a regular BERT-alike model, but it is uniquely capable of embedding positional information about the text alongside the text itself.
We have fine-tuned the model on the DocVQA [2] dataset, far surpassing the current SOTA (all-mpnet-base-v2) [3]:
| Model | HR@3 | HR@5 | HR@10 |
|-------|------|------|-------|
| all-mpnet-base-v2 (Baseline) | 0.2505 | 0.2941 | 0.3624 |
| LayoutLM (Our Model) | 0.3159 | 0.3909 | 0.5019 |
| Relative Improvement | +26.1% | +32.9% | +38.5% |
### Usage
Please refer to the Colab workbook or the blog post to learn more!
### Get in touch
Reach out to [borys.nadykto@bynesoft.com](mailto:borys.nadykto@bynesoft.com) if you'd like help with deploying the model in commerical setting.