Repository: microsoft/graphrag

Issue:
[Bug]: The gleaning is *not* including the original input
### Describe the bug

![image](https://github.com/user-attachments/assets/4e08096e-30ba-4cb3-87f8-baf746f2a3a6)

If you look at the below function:

https://github.com/microsoft/graphrag/blob/309abc982f158c38099c6098d30b35a20972d258/graphrag/index/graph/extractors/graph/graph_extractor.py#L148C5-L182C23
```
async def _process_document(
        self, text: str, prompt_variables: dict[str, str]
    ) -> str:
        response = await self._llm(
            self._extraction_prompt,
            variables={
                **prompt_variables,
                self._input_text_key: text,
            },
        )
        results = response.output or ""

        # Repeat to ensure we maximize entity count
        for i in range(self._max_gleanings):
            glean_response = await self._llm(
                CONTINUE_PROMPT,
                name=f"extract-continuation-{i}",
                history=response.history or [],
            )
            results += glean_response.output or ""

            # if this is the final glean, don't bother updating the continuation flag
            if i >= self._max_gleanings - 1:
                break

            continuation = await self._llm(
                LOOP_PROMPT,
                name=f"extract-loopcheck-{i}",
                history=glean_response.history or [],
                model_parameters=self._loop_args,
            )
            if continuation.output != "YES":
                break

        return results
```

The call to do gleaning does not include the original input 

```
glean_response = await self._llm(
                CONTINUE_PROMPT,
                name=f"extract-continuation-{i}",
                history=response.history or [],
            )
```
`response.history` only includes the last output from the LLM i.e. it is missing the `GRAPH_EXTRACTION_PROMPT` prompt. 

I have verified this by looking at the exchange in the debugger as well. 

Is this expected implementation? I would have thought that in order to glean at minimum the original text (chunk) would be required. Here the gleaning seems to only use the `last response` 

Please guide. Thanks.


### Steps to reproduce

_No response_

### Expected Behavior

_No response_

### GraphRAG Config Used

_No response_

### Logs and screenshots

_No response_

### Additional Information

- GraphRAG Version: 
- Operating System:
- Python Version:
- Related Issues:


Discussion:

Patch:
diff --git a/.semversioner/next-release/patch-20240726142042913643.json b/.semversioner/next-release/patch-20240726142042913643.json
new file mode 100644
index 00000000..4413e7ff
--- /dev/null
+++ b/.semversioner/next-release/patch-20240726142042913643.json
@@ -0,0 +1,4 @@
+{
+  "type": "patch",
+  "description": "add user prompt to history-tracking llm"
+}
diff --git a/graphrag/llm/openai/openai_history_tracking_llm.py b/graphrag/llm/openai/openai_history_tracking_llm.py
index 26ac8e22..77a8ac06 100644
--- a/graphrag/llm/openai/openai_history_tracking_llm.py
+++ b/graphrag/llm/openai/openai_history_tracking_llm.py
@@ -34,5 +34,9 @@ class OpenAIHistoryTrackingLLM(LLM[CompletionInput, CompletionOutput]):
         return LLMOutput(
             output=output.output,
             json=output.json,
-            history=[*history, {"role": "system", "content": output.output}],
+            history=[
+                *history,
+                {"role": "user", "content": input},
+                {"role": "system", "content": output.output},
+            ],
         )


Test patch:
diff --git a/tests/unit/llm/__init__.py b/tests/unit/llm/__init__.py
new file mode 100644
index 00000000..0a3e38ad
--- /dev/null
+++ b/tests/unit/llm/__init__.py
@@ -0,0 +1,2 @@
+# Copyright (c) 2024 Microsoft Corporation.
+# Licensed under the MIT License
diff --git a/tests/unit/llm/openai/__init__.py b/tests/unit/llm/openai/__init__.py
new file mode 100644
index 00000000..0a3e38ad
--- /dev/null
+++ b/tests/unit/llm/openai/__init__.py
@@ -0,0 +1,2 @@
+# Copyright (c) 2024 Microsoft Corporation.
+# Licensed under the MIT License
diff --git a/tests/unit/llm/openai/test_history_tracking_llm.py b/tests/unit/llm/openai/test_history_tracking_llm.py
new file mode 100644
index 00000000..654d0271
--- /dev/null
+++ b/tests/unit/llm/openai/test_history_tracking_llm.py
@@ -0,0 +1,32 @@
+# Copyright (c) 2024 Microsoft Corporation.
+# Licensed under the MIT License
+"""History-tracking LLM Tests."""
+
+import asyncio
+from typing import cast
+
+from graphrag.llm import CompletionLLM, LLMOutput
+from graphrag.llm.openai.openai_history_tracking_llm import OpenAIHistoryTrackingLLM
+
+
+async def test_history_tracking_llm() -> None:
+    async def mock_responder(input: str, **kwargs: dict) -> LLMOutput:
+        await asyncio.sleep(0.0001)
+        return LLMOutput(output=f"response to [{input}]")
+
+    delegate = cast(CompletionLLM, mock_responder)
+    llm = OpenAIHistoryTrackingLLM(delegate)
+
+    response = await llm("input 1")
+    history: list[dict] = cast(list[dict], response.history)
+    assert len(history) == 2
+    assert history[0] == {"role": "user", "content": "input 1"}
+    assert history[1] == {"role": "system", "content": "response to [input 1]"}
+
+    response = await llm("input 2", history=history)
+    history: list[dict] = cast(list[dict], response.history)
+    assert len(history) == 4
+    assert history[0] == {"role": "user", "content": "input 1"}
+    assert history[1] == {"role": "system", "content": "response to [input 1]"}
+    assert history[2] == {"role": "user", "content": "input 2"}
+    assert history[3] == {"role": "system", "content": "response to [input 2]"}