Repository: deepset-ai/haystack

Issue:
PDF Converter on Russian language
**Question**
Hello! I have a question about PDF converter. As I understand, for now it supports only languages provided by the ISO 639-1 standard. As it says in the documentation, trying to convert PDF file with Russian language, which is not listed in the standard, results in some garbage data in the text field of the returned result with the exception of English fragments, of course. Do I understand it correctly that there's no way to use PDFToTextConverter on Russian language PDFs for now? Or is there some workaround that you know or can think of? Will the supported language list be extended in some near future?



Discussion:
Hey @m1kol , 

Our `PDFToTextConverter` is build upon xpdf, which from my understanding also works for Russian. 
I gave it a quick try and the results did not seem complete gibberish. However, I am able to speak / read Russian. So maybe you want to have a look? If you have an example pdf where the conversion failed, it would be very helpful if you could share this so that we can dig deeper

Code used: 

```
from haystack.file_converter.pdf import PDFToTextConverter

converter = PDFToTextConverter()

doc = converter.convert(file_path="URA.pdf")
print(doc)
``` 
PDF: http://www.esb.co.uk/pdf/URA.pdf 
TXT: 
```
кандидат педагогических наук Н.В. Матвеева

Б87

Брагина, Н.В.
УРА! ‒ Уроки Русского Англоговорящим. Russian for Speakers of English / Н.В. Брагина ‒ М.:
Прондо, 2014. ‒ 134 с.
ISBN 978-5-905463-61-7
«УРА! – Уроки русского англоговорящим» является краткосрочным курсом русского языка как
иностранного. Курс рассчитан на 30 академических часов и предназначен для англоговорящих
учащихся, изучающих русский язык «с нуля».В курсе воплощаются идеи авторской национальноориентированной методики, базирующейся на принципах краткосрочного характера обучения,
минимизации учебного материала, культурологического тематического комментирования на
английском языке, наглядности и индукции, как основного способа подачи материала. Основной
целью курса является культурологическая и языковая подготовка в объеме, необходимом для
совершения туристической поездки в Россию.
Компоненты курса – учебник, рабочая тетрадь и CD – содержат много интересных сравнений
между русской и британской культурами, обычаями и менталитетом. Благодаря тематически
структурированному материалу учащиеся после изучения данного курса смогут разговаривать
на повседневные темы.
УДК 811.111
ББК 81.2Англ-923
Author:
Dr Nadezda Bragina (University of St Andrews)
Reviewers:
Dr L.S. Kriuchkova (Moscow State Pedagogical University, Russia)
Dr E.U. Petrishina (International Institute of Management LINK, Russia)
Dr N.V. Matveeva (International Institute of Management LINK, Russia)
“УРА! – Уроки Русского Англоговорящим. Russian for Speakers of English” is a short-term abinitio course of Russian as a foreign language consisting of a book, a workbook and a CD. The course
of 30 academic hours has been especially designed for English-speaking audience and is based on the
communicative approach to language teaching with wide use of visualization, cultural commentaries,
educational games and songs. The course prepares students for a successful and enjoyable first trip to
Russia where they will be able to communicate in simple everyday situations.

ISBN 978-5-905463-61-7

© Н.В. Брагина, 2014
Introduction
«УРА – уроки русского англоговорящим» is a short-term (30 academic
hours) course of Russian as a foreign language aimed at English-speaking students
learning the language from scratch in order to travel to Russia.
The course is based on the communicative approach to language teaching with
wide use of computer technologies, educational games and songs. The teaching
materials produced in a form of a manual, a student book and a CD are enriched with
interesting comparisons of Russian and British culture, customs and mentality. All
language material is thematically structured and covers the situations, vocabulary and
grammar required to enable the students to make basic conversations on everyday
topics.
Every unit begins with a text on Russian everyday culture to help the students
understand Russian reality and people’s behavior better. The main aim of these texts is
to prevent or to reduce cultural shock when the students find themselves in the new
country. The educational process goes from separate lexical items and grammar points
to their use in thematically-determined dialogues. New material is introduced with
multiple examples and pictures to ease the understanding, then it is drilled in simple
exercises. When the students feel comfortable with the material they can move to
dialogues which contain main lexical and grammar patterns of the unit. There is a list
of new words at the end of every unit and a cumulative Russian-English vocabulary at
the back of the book. Also at the end of the book there is a grammar summary which
presents all grammatical forms dealt with in the course. The recordings will not only
help the students with pronunciation but also develop oral and aural skills. The
symbol  indicates which material is on the recordings.
This course has been successfully taught in Belfast Metropolitan College since
2009 and in Dundee Russian School since 2012.
I would like to take this opportunity to express my sincere gratitude to Dr
Ludmila Kriuchkova for her eternal stream of interesting ideas; to John Langran and
Cheryl Johnston for proofreading the text of the book; to Dr Irina Koukoushkina for
being the first person who introduced me to the educational games and creative ways
of language teaching; to Dr Alexander Galkin for recording the dialogues and the
songs of the course with me; to my colleagues and friends who expressed their honest
opinion on the course; to my family for their love and support, and to the students of
Belfast Metropolitan College who were the first to study the language with the course
«УРА – уроки русского англоговорящим».
Dr Nadezda Bragina
May 2012

3
Lesson

Russian Culture

1.Introduction

About the Russian
Language (ex.I,II)

Listening
Song «Алфавит»
(ex. VII)
-Song «Здравствуйте»
(ex.VII,VIII)
-Poem (ex.X)
-Dialogue «First
meeting» (ex. XVI)

2.Getting to know each
other

Social etiquette rules
(ex.I, II)

Basic phrases and
questions to introduce
yourself and get to know
someone
(ex.IV, XIII, XIV, XVIII)

3.My home

Housing in Russia
(ex.I,II)

How to describe a
house/a flat (ex.V,VII), a
room (ex.VIII,X,XVI,
XVII)

Song «Новоселье»
(ex.XVIII)

4.My day

A day in the life of a
Russian person
(ex.I,II)

-How to speak about the
things you usually do
(ex.XIV,XVI)
-How to tell the time
(ex.XIII)

Text «Elena’s day»
(ex.XI)

5.In the town

Moscow (ex.I,II)

How to ask for and give
directions(ex.X,XI)

6.In the restaurant

Russian cuisine and
feasting (ex.I,II)

How to order meals and
drinks in a restaurant
(ex.XII, XIII,XIV)

-Song «Диета»
(ex.X)
-Dialogue «In a
restaurant» (ex.XII)

7.Health

About health
(ex.I,II)

How to ask people how
they feel and say how you
feel (ex.VII,VIIIb,
X, XI,XII)

Dialogue «At the
doctor’s»
(ex.X)

8. How people look

What a Russian person
looks like
(ex.I,II)

How to describe people
(ex.VI,VII,XI)

Song «Русская краса»
(ex.XIII)

9. The weather

The weather in Russia
(ex.I,II)

How to talk about the
weather (ex.VII,XII)

The weather forecast
(ex.VI)

10.My family

A Russian family (ex.I,II)

-How to talk about your
family (ex.V,VI)
-When did you last...?
(ex.XIII)

Text «My grandfather»
(ex. VIIIa)

11. In a souvenir shop

Russian souvenirs
(ex.I,II)

-How to ask the price
(ex.VII)
-Conversations in the
shop (ex.XI)

Dialogue «In a gift shop»
(ex.IX)

12. Transport

Transport
(ex.I,II)

How to request and give
information about
travelling(ex.VIII,X,IX)

Dialogue «How can I get
to Arbat?» (ex.IX)

Appendix 1 – Grammar commentaries
Appendix 2 – Dictionary
Appendix 3 –Listening tasks (transcript)

4

Speaking
Кто это? Who is it?
Что это?What is it?
Это... This is/these are...
(ex.V)

[...]
```
Hello @tholor, thanks for the quick reply!

I've just tried to convert files again and got same gibberish results. I could not attach the files here because of their size so here are links to them:
* [File 1](https://download.schneider-electric.com/files?p_enDocType=Catalog&p_File_Name=%D0%9A%D0%B0%D[…]%BE%D0%B3_Trihal_2016.pdf&p_Doc_Ref=MKP-CAT-TRIHAL-16)
* [File 2](https://download.schneider-electric.com/files?p_enDocType=Catalog&p_File_Name=Panorama_SE_2019_web.pdf&p_Doc_Ref=MKP-CAT-TECHDIG-19)
* [File 3](https://download.schneider-electric.com/files?p_enDocType=Catalog&p_File_Name=Cat_Acti+9_%D0[…]0%D0%BD%D0%B8%D0%B5_19.pdf&p_Doc_Ref=MKP-CAT-ACTI9-19)

At the same time, when I ran a `pdftotext` library on them I got decent results (with the exception of proper table data extraction, of course). 

I ran the code on Google Colab following a link to your first Basic QA Pipeline tutorial and basically did the same thing as you. I think it worth mentioning that when I first tried to initialize PDFToText model, I got the following error:
 `Exception: pdftotext is not installed. It is part of xpdf or poppler-utils software suite.`
The error/help string suggests running
`wget --no-check-certificate https://dl.xpdfreader.com/xpdf-tools-linux-4.02.tar.gz && tar -xvf xpdf-tools-linux-4.02.tar.gz && sudo cp xpdf-tools-linux-4.02/bin64/pdftotext /usr/local/bin`,
but it results in an error, 404 in particular. It turns out that `XpdfReader` has updated and the latest version is 4.03 instead of 4.02. Changing version in the above command makes everything work fine so you should probably update it.
-using pdftotext version 4.02 and haystack 07.0-
I'm facing a similar issue with German characters in pdf files. German Umlaute such as "ü", "ß" etc, are missing after running `convert_files_to_dicts` and the `PDFToTextConverter()`. 
However, when running `pdftotext -enc UTF-8 path_to_my_file.pdf` the resulting txt-file looked good. So is there a way to set the encoding in the file_converter?

Ok, I will have another look. Might be related to the encoding used when storing the text ...

Resolving patch:
diff --git a/haystack/file_converter/pdf.py b/haystack/file_converter/pdf.py
--- a/haystack/file_converter/pdf.py
+++ b/haystack/file_converter/pdf.py
@@ -28,8 +28,8 @@ def __init__(self, remove_numeric_tables: Optional[bool] = False, valid_language
                 """pdftotext is not installed. It is part of xpdf or poppler-utils software suite.
                 
                    Installation on Linux:
-                   wget --no-check-certificate https://dl.xpdfreader.com/xpdf-tools-linux-4.02.tar.gz &&
-                   tar -xvf xpdf-tools-linux-4.02.tar.gz && sudo cp xpdf-tools-linux-4.02/bin64/pdftotext /usr/local/bin
+                   wget --no-check-certificate https://dl.xpdfreader.com/xpdf-tools-linux-4.03.tar.gz &&
+                   tar -xvf xpdf-tools-linux-4.03.tar.gz && sudo cp xpdf-tools-linux-4.03/bin64/pdftotext /usr/local/bin
                    
                    Installation on MacOS:
                    brew install xpdf
@@ -40,14 +40,22 @@ def __init__(self, remove_numeric_tables: Optional[bool] = False, valid_language
 
         super().__init__(remove_numeric_tables=remove_numeric_tables, valid_languages=valid_languages)
 
-    def convert(self, file_path: Path, meta: Optional[Dict[str, str]] = None) -> Dict[str, Any]:
+    def convert(self, file_path: Path, meta: Optional[Dict[str, str]] = None, encoding: str = "Latin1") -> Dict[str, Any]:
         """
-        Extract text from a .pdf file.
+        Extract text from a .pdf file using the pdftotext library (https://www.xpdfreader.com/pdftotext-man.html)
 
         :param file_path: Path to the .pdf file you want to convert
+        :param meta: Optional dictionary with metadata that shall be attached to all resulting documents.
+                     Can be any custom keys and values.
+        :param encoding: Encoding that will be passed as -enc parameter to pdftotext. "Latin 1" is the default encoding
+                         of pdftotext. While this works well on many PDFs, it might be needed to switch to "UTF-8" or
+                         others if your doc contains special characters (e.g. German Umlauts, Cyrillic characters ...).
+                         Note: With "UTF-8" we experienced cases, where a simple "fi" gets wrongly parsed as
+                         "xef\xac\x81c" (see test cases). That's why we keep "Latin 1" as default here.
+                         (See list of available encodings by running `pdftotext -listencodings` in the terminal)
         """
 
-        pages = self._read_pdf(file_path, layout=False)
+        pages = self._read_pdf(file_path, layout=False, encoding=encoding)
 
         cleaned_pages = []
         for page in pages:
@@ -89,7 +97,7 @@ def convert(self, file_path: Path, meta: Optional[Dict[str, str]] = None) -> Dic
         document = {"text": text, "meta": meta}
         return document
 
-    def _read_pdf(self, file_path: Path, layout: bool) -> List[str]:
+    def _read_pdf(self, file_path: Path, layout: bool, encoding: str) -> List[str]:
         """
         Extract pages from the pdf file at file_path.
 
@@ -98,9 +106,9 @@ def _read_pdf(self, file_path: Path, layout: bool) -> List[str]:
                        the content stream order.
         """
         if layout:
-            command = ["pdftotext", "-layout", str(file_path), "-"]
+            command = ["pdftotext", "-enc", encoding, "-layout", str(file_path), "-"]
         else:
-            command = ["pdftotext", str(file_path), "-"]
+            command = ["pdftotext", "-enc", encoding, str(file_path), "-"]
         output = subprocess.run(command, stdout=subprocess.PIPE, shell=False)
         document = output.stdout.decode(errors="ignore")
         pages = document.split("\f")


Test patch:
diff --git a/test/test_pdf_conversion.py b/test/test_pdf_conversion.py
--- a/test/test_pdf_conversion.py
+++ b/test/test_pdf_conversion.py
@@ -28,7 +28,9 @@ def test_table_removal(Converter, xpdf_fixture):
     assert "54x growth" not in pages[0]
 
     # assert text is retained from the document.
-    assert "Adobe Systems made the PDF specification available free of charge in 1993." in pages[0].replace("\n", "")
+    # As whitespace can differ (\n," ", etc.), we standardize all to simple whitespace
+    page_standard_whitespace = " ".join(pages[0].split())
+    assert "Adobe Systems made the PDF specification available free of charge in 1993." in page_standard_whitespace
 
 
 @pytest.mark.tika