Repository: deepset-ai/haystack

Issue:
QuestionGenerator generates wrong document questions for non-default `num_queries_per_doc`
**Describe the bug**
QuestionGenerator, for the non-default values of `num_queries_per_doc` parameter, generates questions that don't match the order of given documents. For example:

```
question_generator = QuestionGenerator(model_name_or_path="valhalla/t5-small-e2e-qg", num_queries_per_doc=2)
questions = question_generator.generate_batch(texts=["Some document text..", "Another document text..."])

assert isinstance(questions, list)
assert len(questions) == 2
assert isinstance(questions[0], list)
assert isinstance(questions[1], list)
assert len(questions[0]) > 0
assert len(questions[1]) > 0
```

However, not all the questions[0] and questions[1] are guaranteed to come from the first and second documents, respectively, but are likely to be mixed up. For example, questions[0] and questions[1] are likely to contain the questions from the first documents. This mixup then spills over into the next item in the questions list, and so on.


**Error message**
Unfortunately, no error is thrown; the errors are silent.

**Expected behavior**
Generated question buckets should match the originating documents.

**Additional context**
None

**To Reproduce**
See `test_questions_generator.py` and `test_qa_generator_no_default_params` method 

**FAQ Check**
- [ ] Have you had a look at [our new FAQ page](https://haystack.deepset.ai/overview/faq)?

**System:**
 - OS: 
 - GPU/CPU:
 - Haystack version (commit or version number): 
 - DocumentStore:
 - Reader:
 - Retriever:



Discussion:

Resolving patch:
diff --git a/haystack/nodes/question_generator/question_generator.py b/haystack/nodes/question_generator/question_generator.py
--- a/haystack/nodes/question_generator/question_generator.py
+++ b/haystack/nodes/question_generator/question_generator.py
@@ -60,6 +60,11 @@ def __init__(
         :param model_version: The version of model to use from the HuggingFace model hub. Can be tag name, branch name, or commit hash.
         :param use_gpu: Whether to use GPU or the CPU. Falls back on CPU if no GPU is available.
         :param batch_size: Number of documents to process at a time.
+        :param num_queries_per_doc: Number of questions to generate per document. However, this is actually a number
+                                    of question to generate per split in the document where the `split_length` determines
+                                    the length of the split and the `split_overlap` determines the overlap between splits.
+                                    Therefore, this parameter is multiplied by the resulting number of splits to get the
+                                    total number of questions generated per document. This value is capped at 3.
         :param progress_bar: Whether to show a tqdm progress bar or not.
         :param use_auth_token: The API token used to download private models from Huggingface.
                                If this parameter is set to `True`, then the token generated when running
@@ -93,7 +98,7 @@ def __init__(
         self.split_overlap = split_overlap
         self.preprocessor = PreProcessor()
         self.prompt = prompt
-        self.num_queries_per_doc = num_queries_per_doc
+        self.num_queries_per_doc = min(num_queries_per_doc, 3)
         self.batch_size = batch_size
         self.sep_token = self.tokenizer.sep_token or sep_token
         self.progress_bar = progress_bar
@@ -237,7 +242,7 @@ def generate_batch(
         left_idx = 0
         right_idx = 0
         for number in number_of_splits:
-            right_idx = left_idx + number
+            right_idx = left_idx + number * self.num_queries_per_doc
             grouped_predictions_split.append(all_string_outputs[left_idx:right_idx])
             left_idx = right_idx
         # Group predictions together by doc list


Test patch:
diff --git a/test/nodes/test_question_generator.py b/test/nodes/test_question_generator.py
--- a/test/nodes/test_question_generator.py
+++ b/test/nodes/test_question_generator.py
@@ -1,3 +1,5 @@
+from typing import List
+
 import pytest
 
 from haystack.pipelines import (
@@ -9,9 +11,68 @@
 from haystack.schema import Document
 
 
-text = 'The Living End are an Australian punk rockabilly band from Melbourne, formed in 1994. Since 2002, the line-up consists of Chris Cheney (vocals, guitar), Scott Owen (double bass, vocals), and Andy Strachan (drums). The band rose to fame in 1997 after the release of their EP Second Solution / Prisoner of Society, which peaked at No. 4 on the Australian ARIA Singles Chart. They have released eight studio albums, two of which reached the No. 1 spot on the ARIA Albums Chart: The Living End (October 1998) and State of Emergency (February 2006). They have also achieved chart success in the U.S. and the United Kingdom. The Band was nominated 27 times and won five awards at the Australian ARIA Music Awards ceremonies: "Highest Selling Single" for Second Solution / Prisoner of Society (1998), "Breakthrough Artist – Album" and "Best Group" for The Living End (1999), as well as "Best Rock Album" for White Noise (2008) and The Ending Is Just the Beginning Repeating (2011). In October 2010, their debut album was listed in the book "100 Best Australian Albums". Australian musicologist Ian McFarlane described the group as "one of Australia’s premier rock acts. By blending a range of styles (punk, rockabilly and flat out rock) with great success, The Living End has managed to produce anthemic choruses and memorable songs in abundance".'
+text = (
+    "The Living End are an Australian punk rockabilly band from Melbourne, formed in 1994. Since 2002, "
+    "the line-up consists of Chris Cheney (vocals, guitar), Scott Owen (double bass, vocals), and Andy "
+    "Strachan (drums). The band rose to fame in 1997 after the release of their EP Second Solution / Prisoner "
+    "of Society, which peaked at No. 4 on the Australian ARIA Singles Chart. They have released eight studio "
+    "albums, two of which reached the No. 1 spot on the ARIA Albums Chart: The Living End (October 1998) and "
+    "State of Emergency (February 2006). They have also achieved chart success in the U.S. and the United "
+    "Kingdom. The Band was nominated 27 times and won five awards at the Australian ARIA Music Awards "
+    'ceremonies: "Highest Selling Single" for Second Solution / Prisoner of Society (1998), "Breakthrough '
+    'Artist – Album" and "Best Group" for The Living End (1999), as well as "Best Rock Album" for White '
+    "Noise (2008) and The Ending Is Just the Beginning Repeating (2011). In October 2010, their debut album "
+    'was listed in the book "100 Best Australian Albums". Australian musicologist Ian McFarlane described '
+    'the group as "one of Australia’s premier rock acts. By blending a range of styles (punk, rockabilly '
+    "and flat out rock) with great success, The Living End has managed to produce anthemic choruses and "
+    'memorable songs in abundance".'
+)
 document = Document(content=text)
 query = "Living End"
+keywords = [
+    "Australian",
+    "punk",
+    "drummer",
+    "Living",
+    "band",
+    "Band",
+    "Second",
+    "album",
+    "albums",
+    "dialect",
+    "music",
+    "book",
+    "group",
+    "produce",
+    "Music",
+    "Awards",
+    "year",
+    "released",
+]
+text_2 = (
+    "Berlin straddles the banks of the Spree, which flows into the Havel (a tributary of the Elbe) in the "
+    "western borough of Spandau. Among the city's main topographical features are the many lakes in the western "
+    "and southeastern boroughs formed by the Spree, Havel and Dahme, the largest of which is Lake Müggelsee. "
+    "Due to its location in the European Plain, Berlin is influenced by a temperate seasonal climate. About "
+    "one-third of the city's area is composed of forests, parks, gardens, rivers, canals and lakes. The city "
+    "lies in the Central German dialect area, the Berlin dialect being a variant of the Lusatian-New Marchian "
+    "dialects."
+)
+document_2 = Document(content=text_2)
+keywords_2 = [
+    "Berlin",
+    "Elbe",
+    "Spandau",
+    "Spree",
+    "boroughs",
+    "lakes",
+    "largest",
+    "seasonal",
+    "climate",
+    "city",
+    "dialect",
+    "German",
+]
 
 
 def test_qg_pipeline(question_generator):
@@ -22,6 +83,47 @@ def test_qg_pipeline(question_generator):
     assert len(result["generated_questions"][0]["questions"]) > 0
 
 
+def test_qg_pipeline_non_default_params():
+    question_generator = QuestionGenerator(model_name_or_path="valhalla/t5-small-e2e-qg", num_queries_per_doc=2)
+    p = QuestionGenerationPipeline(question_generator)
+    result = p.run(documents=[document, document_2])
+    assert isinstance(result, dict)
+    assert "generated_questions" in result
+    assert "documents" in result
+    assert isinstance(result["generated_questions"], list)
+    assert isinstance(result["documents"], list)
+    assert len(result["generated_questions"]) == 2
+    assert len(result["documents"]) == 2
+    assert len(result["generated_questions"][0]["questions"]) == 26
+    assert len(result["generated_questions"][1]["questions"]) == 12
+
+    # first list of questions should be about Australian punk band
+    verify_questions(result["generated_questions"][0]["questions"], keywords)
+    # second list of questions should be about Berlin
+    verify_questions(result["generated_questions"][1]["questions"], keywords_2)
+
+
+@pytest.mark.parametrize("split_length, num_queries_per_doc", [(50, 1), (50, 2), (50, 3), (100, 1), (100, 2), (100, 3)])
+def test_qa_generator_non_default_params(split_length, num_queries_per_doc):
+    question_generator = QuestionGenerator(
+        model_name_or_path="valhalla/t5-small-e2e-qg",
+        split_length=split_length,
+        num_queries_per_doc=num_queries_per_doc,
+    )
+    questions = question_generator.generate_batch(texts=[document.content, document_2.content])
+    assert isinstance(questions, list)
+    assert len(questions) == 2
+    assert isinstance(questions[0], list)
+    assert isinstance(questions[1], list)
+    assert len(questions[0]) > 0
+    assert len(questions[1]) > 0
+
+    # first list of questions should be about Australian punk band
+    verify_questions(questions[0], keywords)
+    # second list of questions should be about Berlin
+    verify_questions(questions[1], keywords_2)
+
+
 @pytest.mark.parametrize("retriever,document_store", [("tfidf", "memory")], indirect=True)
 def test_rqg_pipeline(question_generator, retriever):
     retriever.document_store.write_documents([document])
@@ -42,3 +144,8 @@ def test_qag_pipeline(question_generator, reader):
     assert len(results["queries"]) == len(results["answers"])
     assert len(results["answers"]) > 0
     assert results["answers"][0][0].answer is not None
+
+
+def verify_questions(questions: List[str], question_keywords: List[str]):
+    for q in questions:
+        assert any(word in q for word in question_keywords)