Package: wagtail-textract
Text extraction for Wagtail document searchThis package is for replacingWagtail's Document class with one
that allows searching in Document file contents usingtextract.Textract can extract text from (amongothers) PDF, Excel and Word files.The package was inspired by the"Search: Extract text from documents" issuein Wagtail.Documents will work as before, except that Document search in Wagtail's admin interface
will also find search terms in the files' contents.Some screenshots to illustrate.In our fresh Wagtail site withwagtail_textractinstalled,
we uploaded afile calledtest_document.pdfwith handwritten text in it.
It is listed in the admin interface under Documents:If we now search in Documents for the wordcorrect, which is one of the handwritten words,
the live search finds it:The assumption is that this search should not only be available in Wagtail's admin interface,
but also in a public-facing search view, for which we provide a code example.RequirementsWagtail 2 (seetox.ini)TheTextract dependenciesMaturityWe have been using this package in production since August 2018 onhttps://nuffic.nl.InstallationInstall theTextract dependenciesAddwagtail_textractto your requirements and/orpip install wagtail_textractAdd to your DjangoINSTALLED_APPS.PutWAGTAILDOCS_DOCUMENT_MODEL = "wagtail_textract.document"in your Django settings.Note: You'll get an incompatibility warning during installation of wagtail_textract (Wagtail 2.0.1 installed):requests 2.18.4 has requirement chardet<3.1.0,>=3.0.2, but you'll have chardet 2.3.0 which is incompatible.
textract 1.6.1 has requirement beautifulsoup4==4.5.3, but you'll have beautifulsoup4 4.6.0 which is incompatible.We haven't seen this leading to problems, but it's something to keep in mind.TesseractIn order to maketextractuseTesseract, which happens if regulartextractfinds no text, you need to add the data files that Tesseract can
base its word matching on.Create atessdatadirectory in your project directory, and download thelanguagesyou want.TranscribingTranscription is done automatically after Document save,
in anasyncioexecutor to prevent blocking the response during processing.To transcribe all existing Documents, run the management command::./manage.py transcribe_documentsThis may take a long time, obviously.Usage in custom viewHere is a code example for a search view (outside Wagtail's admin interface)
that shows both Page and Document results.fromitertoolsimportchainfromwagtail.core.modelsimportPagefromwagtail.documents.modelsimportget_document_modeldefsearch(request):# Searchsearch_query=request.GET.get('query',None)ifsearch_query:page_results=Page.objects.live().search(search_query)document_results=Document.objects.search(search_query)search_results=list(chain(page_results,document_results))# Log the query so Wagtail can suggest promoted resultsQuery.get(search_query).add_hit()else:search_results=Page.objects.none()# Render templatereturnrender(request,'website/search_results.html',{'search_query':search_query,'search_results':search_results,})Your template should allow for handling Documents differently than Pages,
because you can't dopageurl resulton a Document:{% if result.file %}
   <a href="{{ result.url }}">{{ result }}</a>
{% else %}
   <a href="{% pageurl result %}">{{ result }}</a>
{% endif %}What if you already use a custom Document model?In order to use wagtail_textract, yourCustomizedDocumentmodel should do
the same aswagtail_textract's Document:subclassTranscriptionMixinaltersearch_fieldsfromwagtail_textract.modelsimportTranscriptionMixinclassCustomizedDocument(TranscriptionMixin,...):"""Extra fields and methods for Document model."""search_fields=...+[index.SearchField('transcription',partial_match=False,),]Note that the first class to subclass should beTranscriptionMixin,
so itssave()takes precedence over that of the other parent classes.TestsTo run tests, checkout this repository and:make testCoverageA coverage report will be generated in./coverage_html_report/.ContributorsKarl HobleyBertrand BordageKees HinkTom HendrikxCoen van der KampMike OverkampThibaud ColasDan BraghisDan Swain