Repository: langchain-ai/langchain (Python)
Date: 2024-04-05 12:56:34+00:00
Task type: Bug Fix

Issue:
BooleanOutputParser expected output value error
### System Info

Hi, I am using LLMChainFilter.from_llm(llm) but while running, I am getting this error:

ValueError: BooleanOutputParser expected output value to either be YES or NO. Received Yes, the context is relevant to the question as it provides information about the problem in the.

How do I resolve this error?

Langchain version: 0.0.308

### Who can help?

@agola11

### Information

- [X] The official example notebooks/scripts
- [ ] My own modified scripts

### Related Components

- [ ] LLMs/Chat Models
- [ ] Embedding Models
- [ ] Prompts / Prompt Templates / Prompt Selectors
- [ ] Output Parsers
- [ ] Document Loaders
- [ ] Vector Stores / Retrievers
- [ ] Memory
- [ ] Agents / Agent Executors
- [ ] Tools / Toolkits
- [ ] Chains
- [ ] Callbacks/Tracing
- [ ] Async

### Reproduction

from langchain.retrievers import ContextualCompressionRetriever
from langchain.retrievers.document_compressors import LLMChainExtractor, LLMChainFilter

llm = SageMakerEndpointModel
_filter = LLMChainFilter.from_llm(llm)
compressor = LLMChainExtractor.from_llm(llm)
compression_retriever = ContextualCompressionRetriever(base_compressor=_filter, base_retriever=faiss_retriever)

compressed_docs = compression_retriever.get_relevant_documents("What did the president say about Ketanji Jackson Brown?")


### Expected behavior

Get filtered docs


Issue discussion:
_🤖_
<!-- Answer -->
Hello,

The error you're encountering is due to the output of your language model (LLM) not matching the expected values "YES" or "NO" that the `BooleanOutputParser` class is designed to handle. 

The `BooleanOutputParser` class in the LangChain codebase is used to parse the output of a language model call to a boolean value. It takes the output of a language model, represented as a string, and converts it into a boolean value. The class has two string attributes, `true_val` and `false_val`, which represent the string values that should be parsed as `True` and `False` respectively. By default, these are set to "YES" and "NO". 

Here's the relevant code that's causing the error:

```py
cleaned_text = text.strip()
if cleaned_text.upper() not in (self.true_val.upper(), self.false_val.upper()):
    raise ValueError(
        f"BooleanOutputParser expected output value to either be "
        f"{self.true_val} or {self.false_val}. Received {cleaned_text}."
    )
```

This code is checking if the cleaned, uppercased output of your LLM (`cleaned_text.upper()`) is either "YES" or "NO". If it's not, it raises a `ValueError`. So, you need to ensure that your LLM's output matches these expectations.

To resolve this issue, you need to ensure that the output from your LLM is either "YES" or "NO". If your LLM is producing different outputs, you might need to adjust your LLM or create a custom output parser that can handle the specific outputs of your LLM.

In the context shared, the `LLMChainFilter` class uses an LLM to filter documents based on their relevance to a query. The `from_llm` class method creates an `LLMChainFilter` from a language model. If no prompt is provided, it uses a default prompt that expects a boolean output from the LLM. This is where the `BooleanOutputParser` is used, and it expects the output to be either "YES" or "NO".

I hope this helps! If you have any other questions, feel free to ask.

Best,
Dosu

## Sources
- [Issue: value error in BooleanOutputParser](https://github.com/langchain-ai/langchain/issues/5396)
- [libs/langchain/langchain/output_parsers/boolean.py](https://github.com/langchain-ai/langchain/blob/master/libs/langchain/langchain/output_parsers/boolean.py)
- [libs/langchain/langchain/retrievers/document_compressors/chain_filter.py](https://github.com/langchain-ai/langchain/blob/master/libs/langchain/langchain/retrievers/document_compressors/chain_filter.py)
- [libs/langchain/tests/unit_tests/output_parsers/test_boolean_parser.py](https://github.com/langchain-ai/langchain/blob/master/libs/langchain/tests/unit_tests/output_parsers/test_boolean_parser.py)

---
This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.
@ankur287  Looks like the output parser doesn't retry when the LLM output isn't as expected. We should make it more robust by retrying. I can try fixing this issue if that's ok with you.
@agola11 Can you please assign this issue to me? I'm unable to do it myself.
I fixed the issue temporarily by creating a rule that if "YES" in output just return "YES". However, we need to make this change as we cannot expect LLM to always say just yes or no. In my case model said, 'Yes answer is available in context'
@ankur287 Do you mind posting how you were able to get around this issue in detail since LangChain itself hasn't really fixed this? 

If not is there an issue tracking this problem? I have found a quick workaround by implementing my own boolean output parser to default to YES along with checking if YES/NO is in the output instead of strict matching. I am happy to make a PR to address this problem.
I posted above how I fixed it. See my last comment

Solution patch:
diff --git a/libs/langchain/langchain/output_parsers/boolean.py b/libs/langchain/langchain/output_parsers/boolean.py
--- a/libs/langchain/langchain/output_parsers/boolean.py
+++ b/libs/langchain/langchain/output_parsers/boolean.py
@@ -1,3 +1,5 @@
+import re
+
 from langchain_core.output_parsers import BaseOutputParser
 
 
@@ -17,26 +19,31 @@ def parse(self, text: str) -> bool:
 
         Returns:
             boolean
-
         """
-        cleaned_upper_text = text.strip().upper()
-        if (
-            self.true_val.upper() in cleaned_upper_text
-            and self.false_val.upper() in cleaned_upper_text
-        ):
-            raise ValueError(
-                f"Ambiguous response. Both {self.true_val} and {self.false_val} in "
-                f"received: {text}."
-            )
-        elif self.true_val.upper() in cleaned_upper_text:
+        regexp = rf"\b({self.true_val}|{self.false_val})\b"
+
+        truthy = {
+            val.upper()
+            for val in re.findall(regexp, text, flags=re.IGNORECASE | re.MULTILINE)
+        }
+        if self.true_val.upper() in truthy:
+            if self.false_val.upper() in truthy:
+                raise ValueError(
+                    f"Ambiguous response. Both {self.true_val} and {self.false_val} "
+                    f"in received: {text}."
+                )
             return True
-        elif self.false_val.upper() in cleaned_upper_text:
+        elif self.false_val.upper() in truthy:
+            if self.true_val.upper() in truthy:
+                raise ValueError(
+                    f"Ambiguous response. Both {self.true_val} and {self.false_val} "
+                    f"in received: {text}."
+                )
             return False
-        else:
-            raise ValueError(
-                f"BooleanOutputParser expected output value to include either "
-                f"{self.true_val} or {self.false_val}. Received {text}."
-            )
+        raise ValueError(
+            f"BooleanOutputParser expected output value to include either "
+            f"{self.true_val} or {self.false_val}. Received {text}."
+        )
 
     @property
     def _type(self) -> str:


Test patch:
diff --git a/libs/langchain/tests/unit_tests/output_parsers/test_boolean_parser.py b/libs/langchain/tests/unit_tests/output_parsers/test_boolean_parser.py
--- a/libs/langchain/tests/unit_tests/output_parsers/test_boolean_parser.py
+++ b/libs/langchain/tests/unit_tests/output_parsers/test_boolean_parser.py
@@ -1,3 +1,5 @@
+import pytest
+
 from langchain.output_parsers.boolean import BooleanOutputParser
 
 
@@ -24,16 +26,16 @@ def test_boolean_output_parser_parse() -> None:
     result = parser.parse("Not relevant (NO)")
     assert result is False
 
+    # Test valid input
+    result = parser.parse("NOW this is relevant (YES)")
+    assert result is True
+
     # Test ambiguous input
-    try:
-        parser.parse("yes and no")
-        assert False, "Should have raised ValueError"
-    except ValueError:
-        pass
-
-    # Test invalid input
-    try:
-        parser.parse("INVALID")
-        assert False, "Should have raised ValueError"
-    except ValueError:
-        pass
+    with pytest.raises(ValueError):
+        parser.parse("YES NO")
+
+    with pytest.raises(ValueError):
+        parser.parse("NO YES")
+    # Bad input
+    with pytest.raises(ValueError):
+        parser.parse("BOOM")