Repository: NVIDIA/NeMo-Guardrails

Issue:
bug: Enhance LLM response evaluation in generate_value to handle invalid syntax
### Did you check docs and existing issues?

- [x] I have read all the NeMo-Guardrails docs
- [x] I have updated the package to the latest version before submitting this issue
- [ ] (optional) I have used the develop branch
- [x] I have searched the existing issues of NeMo-Guardrails

### Python version (python --version)

Python 3.10.16

### Operating system/version

Windows 11

### NeMo-Guardrails version (if you must use a specific version and not the latest

0.11.0 (latest pip install)

### **Code Location**
- **File**: `NeMo-Guardrails/nemoguardrails/actions/llm/generation.py`
- **Function**: `generate_value`

### Describe the bug

In the current implementation of `generate_value` located at `NeMo-Guardrails/nemoguardrails/actions/llm/generation.py`, the method directly uses `literal_eval` to parse LLM-generated values. This approach can cause runtime errors, such as `SyntaxError`, when the generated values contain unescaped quotes, invalid syntax, or other formatting issues. 



### Steps To Reproduce

1. Use the `generate_value` method in `generation.py` with an LLM-generated response containing unescaped single quotes, e.g., `"It's a sunny day"`. or  `'"It is a sunny day"'` or `"'It is a sunny day'"`
2. Observe the exception raised by `literal_eval`.

### Expected Behavior

The evaluation of the generated value should:
1. Escape special characters (e.g., single and double quotes) gracefully.
2. Avoid raising runtime exceptions by sanitizing the input string.

### Actual Behavior

The original `literal_eval` approach raises `SyntaxError` when the generated value contains unescaped quotes or invalid syntax.

### **Proposed Solution**
To address these issues, I implemented a new `safe_eval` function and updated the `generate_value` method as follows:

#### **Modified Code**
1. **Updated `generate_value`(line1040-1042)**:
   ```python
   log.info(f"Generated value for ${var_name}: {value}")

   try:
       return safe_eval(value)
   except Exception:
       raise Exception(f"Invalid LLM response: `{value}`")
   ```

2. **New `safe_eval` Function**:
   ```python
   def safe_eval(input_value: str) -> str:
       if input_value.startswith(("'", '"')) and input_value.endswith(("'", '"')):
           try:
               return literal_eval(input_value)
           except ValueError:
               pass  
       escaped_value = input_value.replace("'", "\\'").replace('"', '\\"')
       input_value = f"'{escaped_value}'"
       return literal_eval(input_value)
   ```

#### **Functionality of `safe_eval`**:
- **Escaping Special Characters**: Automatically escapes single (`'`) and double (`"`) quotes to avoid syntax errors.
- **Graceful Fallback**: Handles exceptions raised by `literal_eval` and ensures sanitized input.
- **Integration**: Works seamlessly with the `generate_value` method.

Discussion:

Solution patch:
diff --git a/nemoguardrails/actions/llm/generation.py b/nemoguardrails/actions/llm/generation.py
index d8a4cc3c..7bb9f42b 100644
--- a/nemoguardrails/actions/llm/generation.py
+++ b/nemoguardrails/actions/llm/generation.py
@@ -64,7 +64,12 @@ from nemoguardrails.patch_asyncio import check_sync_call_from_async_loop
 from nemoguardrails.rails.llm.config import EmbeddingSearchProvider, RailsConfig
 from nemoguardrails.rails.llm.options import GenerationOptions
 from nemoguardrails.streaming import StreamingHandler
-from nemoguardrails.utils import get_or_create_event_loop, new_event_dict, new_uuid
+from nemoguardrails.utils import (
+    get_or_create_event_loop,
+    new_event_dict,
+    new_uuid,
+    safe_eval,
+)
 
 log = logging.getLogger(__name__)
 
@@ -1039,7 +1044,11 @@ class LLMGenerationActions:
 
         log.info(f"Generated value for ${var_name}: {value}")
 
-        return literal_eval(value)
+        try:
+            return safe_eval(value)
+        except Exception as e:
+            log.error(f"Error evaluating value: {value}. Error: {str(e)}")
+            raise ValueError(f"Invalid LLM response: `{value}`")
 
     @action(is_system_action=True)
     async def generate_intent_steps_message(
diff --git a/nemoguardrails/utils.py b/nemoguardrails/utils.py
index dfa12363..6c84933d 100644
--- a/nemoguardrails/utils.py
+++ b/nemoguardrails/utils.py
@@ -21,6 +21,7 @@ import os
 import random
 import re
 import uuid
+from ast import literal_eval
 from collections import namedtuple
 from datetime import datetime, timezone
 from enum import Enum
@@ -384,3 +385,26 @@ def is_ignored_by_railsignore(filename: str, ignore_patterns: str) -> bool:
             break
 
     return ignore
+
+
+def safe_eval(input_value: str) -> str:
+    """
+    Safely evaluate a string to handle unescaped quotes or invalid syntax from the async generate_value action.
+
+    Args:
+        input_value (str): The input string to evaluate.
+
+    Returns:
+        str: The evaluated and properly formatted string.
+
+    Raises:
+        ValueError: If the input cannot be safely evaluated.
+    """
+    if input_value.startswith(("'", '"')) and input_value.endswith(("'", '"')):
+        try:
+            return literal_eval(input_value)
+        except (ValueError, SyntaxError):
+            pass
+    escaped_value = input_value.replace("'", "\\'").replace('"', '\\"')
+    input_value = f"'{escaped_value}'"
+    return literal_eval(input_value)


Test patch:
diff --git a/tests/test_utils.py b/tests/test_utils.py
index ef58b649..1ce893a2 100644
--- a/tests/test_utils.py
+++ b/tests/test_utils.py
@@ -15,7 +15,7 @@
 
 import pytest
 
-from nemoguardrails.utils import new_event_dict
+from nemoguardrails.utils import new_event_dict, safe_eval
 
 
 def test_event_generation():
@@ -119,3 +119,28 @@ def test_wrong_property_type():
             event_type,
             script=script,
         )
+
+
+@pytest.mark.parametrize(
+    "input_value, expected_result",
+    [
+        ('"It\'s a sunny day"', "It's a sunny day"),  # double quotes with single quote
+        (
+            "\"He said, 'Hello'\"",
+            "He said, 'Hello'",
+        ),  # double quotes with nested single quote
+        (
+            "It's a sunny day",
+            "It's a sunny day",
+        ),  # unquoted string containing single quote
+        (
+            "It is a sunny day",
+            "It is a sunny day",
+        ),  # plain string not wrapped in quotes
+        ("", ""),  # empty string
+    ],
+)
+def test_safe_eval(input_value, expected_result):
+    """Test safe_eval with various input values."""
+    result = safe_eval(input_value)
+    assert result == expected_result