Repository: kharvd/gpt-cli

Issue:
Error on input `--a b`
I have a reproducible error when the bot is given input string `--a b`.
```text
$ gpt
Hi! I'm here to help. Type :q or Ctrl-D to exit, :c or Ctrl-C and Enter to clear
the conversation, :r or Ctrl-R to re-generate the last response. To enter
multi-line mode, enter a backslash \ followed by a new line. Exit the multi-line
mode by pressing ESC and then Enter (Meta+Enter). Try :? for help.
> --a b
Invalid argument: a. Allowed arguments: ['model', 'temperature', 'top_p']
Invalid argument: a. Allowed arguments: ['model', 'temperature', 'top_p']
NoneType: None
```
```text
$ gpt --version
gpt-cli v0.1.3
```

<details>
<summary>
Here is the output from running `$ gpt --log_file log.txt --log_level DEBUG`:
</summary>

```text
$ cat log.txt
2023-07-17 18:26:33,531 - gptcli - INFO - Starting a new chat session. Assistant config: {'messages': [], 'temperature': 0.0, 'model': 'gpt-4'}
2023-07-17 18:26:33,539 - gptcli-session - INFO - Chat started
2023-07-17 18:26:33,539 - asyncio - DEBUG - Using selector: EpollSelector
2023-07-17 18:26:35,314 - gptcli-session - ERROR - Invalid argument: a. Allowed arguments: ['model', 'temperature', 'top_p']
NoneType: None
```
</details>

For context, this error came up when I copy/pasted the following rustc error message into the cli using `multiline>` mode:
```
 15 | ) -> impl Iterator<Item = AdaptedRecord> {
    |      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ `()` is not an iterator
    |
    = help: the trait `Iterator` is not implemented for `()`

 For more information about this error, try `rustc --explain E0277`
```
This resulted in `Invalid argument: explain. Allowed arguments: ...`

Discussion:
@Jasha10: It looks like the root cause is that the [`parse_args`](https://github.com/kharvd/gpt-cli/blob/eb1a4160f4990268f5aa05ea373aa22f034e4e27/gptcli/cli.py#L123-L131) function matches the regex pattern `--(\w+)(?:\s+|=)([^\s]+)` and treats it specially.

@kharvd, I'd like to submit a PR that allows the user to disable this regex matching behavior. Would that be acceptable to you?@kharvd: Thanks for a detailed report. If you'd like to add a config option to disable this (and maybe even make it disabled by default), I'm on board - just want to make sure the option is there, because I use `--model` a lot to run the same prompt with a different model.

One alternative idea is to allow switching the model and other args via `:` commands now that we have `:q` and friends: e.g. `:model gpt-4` or `:temperature 0.0`@Jasha10: > One alternative idea is to allow switching the model and other args via `:` commands

Let me make sure I understand what you have in mind: if you write `:model gpt-4`, would this change the model for all subsequent messages?
I believe the current behavior is that `Message ... --model gpt-4` changes the model to `gpt-4` only for the current message, and subsequent messages will revert back to the model that was configured at the start of the chat session.@wfjt: Was hit this as well by including a `-----BEGIN AGE ENCRYPTED FILE-----\n...` string in multiline mode, got an error about `-BEGIN` being an invalid parameter. Quite annoying having to escape 20 times for the header and footer. I tried `gpt --` but that made no difference, same error.@sghael: related comment:  https://github.com/kharvd/gpt-cli/pull/21#issuecomment-2184151092  
@stevenwalton: The problem is not resolved and the error can still be hit if "--" is not contained in the specified deliminators. 

I was having a lengthy conversation and got
```bash
Invalid argument: grep. Allowed arguments: ['model', 'temperature', 'top_p']
```
`gpt --version` shows `gpt-cli v0.2.0` but I noticed that [the sub-version was not bumped](https://github.com/kharvd/gpt-cli/blob/main/gptcli/__init__.py). 

So instead I'll give you this to verify
`````bash
$ which gpt
/Users/steven/.mamba/bin/gpt
$ cat /Users/steven/.mamba/lib/python3.12/site-packages/gptcli/cli.py | grep "def parse_args" -A 34
def parse_args(input: str) -> Tuple[str, Dict[str, Any]]:
    # Extract parts enclosed in specific delimiters (triple backticks, triple quotes, single backticks)
    extracted_parts = []
    delimiters = ['```', '"""', '`']

    def replacer(match):
        for i, delimiter in enumerate(delimiters):
            part = match.group(i + 1)
            if part is not None:
                extracted_parts.append((part, delimiter))
                break
        return f"__EXTRACTED_PART_{len(extracted_parts) - 1}__"

    # Construct the regex pattern dynamically from the delimiters list
    pattern_fragments = [re.escape(d) + '(.*?)' + re.escape(d) for d in delimiters]
    pattern = re.compile('|'.join(pattern_fragments), re.DOTALL)

    input = pattern.sub(replacer, input)

    # Parse the remaining string for arguments
    args = {}
    regex = r'--(\w+)(?:=(\S+)|\s+(\S+))?'
    matches = re.findall(regex, input)

    if matches:
        for key, value1, value2 in matches:
            value = value1 if value1 else value2 if value2 else ''
            args[key] = value.strip("\"'")
        input = re.sub(regex, "", input).strip()

    # Add back the extracted parts, with enclosing backticks or quotes
    for i, (part, delimiter) in enumerate(extracted_parts):
        input = input.replace(f"__EXTRACTED_PART_{i}__", f"{delimiter}{part.strip()}{delimiter}")

    return input, args
````` 
I imported the above lines into a python terminal and passed my prompt through it and found that `args` captured `{'grep': 'renderer'}`. 
 I was able to verify that the following line, in isolation, gets captured by args.
```
>  run journalctl -b 0 --grep "renderer for":
```
The issue is that I did not surround the command with markdown command backticks. 

This has made me realize that there is a larger parsing issue at hand here.

First, I think the regex is inaccurate. It looks for any case of the flag pattern. But instead, we should look for the flag pattern if it appears at the at the beginning of the prompt. Maybe I'm lacking a bit of imagination, but I do not think you can change the {temperature,model,top_p} mid prompt and am pretty confident this isn't supported. So I think it is fine to force flags to be at the beginning of a prompt. 

```python
pattern = \
    r"(?:--|:)" \            # Find -- or : but don't put in capture group
    r"(\w[^\s=]*)" \         # flag keyword starts with a letter and excludes whitespace or =
    r"(?:[ =])" \            # Deliminator of space or equal sign, not in capture group
    r"(\w[^\s=]*|\d+[.]\d*)" # Flag argument begins with letter and doesn't have space or = or is a number (int or float) 

args = re.findall(pattern,         # get all flag pairs
                  re.match(        # limit matching to sequential flags at beginning of prompt
                      r"^(" \      # starting anchor and open capture group 
                      + pattern \ 
                      + r"\s?)*",  # look for patterns but allow ending in space
                  input
                  ).group()        # converts back to string
)
```
`(?:--|:)(\w[^\s=]*)(?:[ =])(\w[^\s=]*|\d+[.]\d*)` matches anything in the flag formulation with a key/argument pair, so we look for a sequence of flags and argument pairs that only appear at the beginning of the prompt. 

I believe this solves the problem?

I tested my string that failed earlier and the new result is `[]`. Testing the string `"--temperature=5.0 :model GPT-4 --top-p=1 --top_p=5.7 hello world --newflag=boop-de --bad=flag=doop-de-doop"` yields `[('temperature', '5.0'), ('model', 'GPT-4'), ('top-p', '1'), ('top_p', '5.7')]` and the string `"--temperature=5.0 :model GPT-4 --top-p=1 --top_p=5.7 --newflag=boop-de --bad=flag=doop-de-doop hello world"` yields `[('temperature', '5.0'),('model', 'GPT-4'),('top-p', '1'),('top_p', '5.7'),('newflag', 'boop-de'),('bad', 'flag')]`. Also, adding "a" to the beginning of the test strings and causes an empty result.
                    
If I understand things correctly, we can remove all the logic from @sghael, including the deliminators which also allows us to create proper codeblocks with arbitrary number of '`' (as was used for this comment). So I believe we can rewrite as
```python
def parse_args(input: str) -> Tuple[str, Dict[str, Any]]:
    pattern = r"(?:--|:)(\w[^\s=]*)(?:[ =])(\w[^\s=]*|\d+[.]\d*)"
    arg_pattern = r"^(" + pattern + r"\s?)*"
    args = re.findall(pattern, re.match(arg_pattern, input).group())
    args = dict((k,v) for k,v in args)
    input = re.sub(arg_pattern, "", input)
    return input, args
```
Have I missed anything? If not, I have a PR ready to go.@stevenwalton: @kharvd, you need to redo the check for my PR because it got stuck on a build (I added no build files...). It doesn't look like I can cause a rebuild and when I matched to the current branch it doesn't auto-recheck. 

Solution patch:
diff --git a/gptcli/cli.py b/gptcli/cli.py
index d3f2f93..65e1040 100644
--- a/gptcli/cli.py
+++ b/gptcli/cli.py
@@ -1,26 +1,19 @@
 import re
+from typing import Any, Dict, Optional, Tuple
+
+from openai import BadRequestError, OpenAIError
 from prompt_toolkit import PromptSession
 from prompt_toolkit.history import FileHistory
-from openai import OpenAIError, BadRequestError
 from prompt_toolkit.key_binding import KeyBindings, KeyPressEvent
 from prompt_toolkit.key_binding.bindings import named_commands
 from rich.console import Console
 from rich.live import Live
 from rich.markdown import Markdown
-from typing import Any, Dict, Optional, Tuple
-
 from rich.text import Text
-from gptcli.session import (
-    ALL_COMMANDS,
-    COMMAND_CLEAR,
-    COMMAND_QUIT,
-    COMMAND_RERUN,
-    ChatListener,
-    InvalidArgumentError,
-    ResponseStreamer,
-    UserInputProvider,
-)
 
+from gptcli.session import (ALL_COMMANDS, COMMAND_CLEAR, COMMAND_QUIT,
+                            COMMAND_RERUN, ChatListener, InvalidArgumentError,
+                            ResponseStreamer, UserInputProvider)
 
 TERMINAL_WELCOME = """
 Hi! I'm here to help. Type `:q` or Ctrl-D to exit, `:c` or Ctrl-C and Enter to clear
@@ -121,12 +114,38 @@ class CLIChatListener(ChatListener):
 
 
 def parse_args(input: str) -> Tuple[str, Dict[str, Any]]:
+    # Extract parts enclosed in specific delimiters (triple backticks, triple quotes, single backticks)
+    extracted_parts = []
+    delimiters = ['```', '"""', '`']
+
+    def replacer(match):
+        for i, delimiter in enumerate(delimiters):
+            part = match.group(i + 1)
+            if part is not None:
+                extracted_parts.append((part, delimiter))
+                break
+        return f"__EXTRACTED_PART_{len(extracted_parts) - 1}__"
+
+    # Construct the regex pattern dynamically from the delimiters list
+    pattern_fragments = [re.escape(d) + '(.*?)' + re.escape(d) for d in delimiters]
+    pattern = re.compile('|'.join(pattern_fragments), re.DOTALL)
+
+    input = pattern.sub(replacer, input)
+
+    # Parse the remaining string for arguments
     args = {}
-    regex = r"--(\w+)(?:\s+|=)([^\s]+)"
+    regex = r'--(\w+)(?:=(\S+)|\s+(\S+))?'
     matches = re.findall(regex, input)
+
     if matches:
-        args = dict(matches)
-        input = input.split("--")[0].strip()
+        for key, value1, value2 in matches:
+            value = value1 if value1 else value2 if value2 else ''
+            args[key] = value.strip("\"'")
+        input = re.sub(regex, "", input).strip()
+
+    # Add back the extracted parts, with enclosing backticks or quotes
+    for i, (part, delimiter) in enumerate(extracted_parts):
+        input = input.replace(f"__EXTRACTED_PART_{i}__", f"{delimiter}{part.strip()}{delimiter}")
 
     return input, args
 


Test patch:
diff --git a/tests/test_term_utils.py b/tests/test_term_utils.py
index 511a4c8..fee724e 100644
--- a/tests/test_term_utils.py
+++ b/tests/test_term_utils.py
@@ -16,3 +16,60 @@ def test_parse_args():
         "this is a prompt",
         {"bar": "1.0", "baz": "2.0"},
     )
+    assert parse_args("this is a prompt --bar 1.0") == (
+        "this is a prompt",
+        {"bar": "1.0"},
+    )
+
+
+def test_parse_with_escape_blocks():
+    test_cases = [
+        (
+            # escaped text at end of prompt
+            "this is a prompt --bar=1.0 {start}--baz=2.0{end}",
+            "this is a prompt  {start}--baz=2.0{end}",
+            {"bar": "1.0"},
+        ),
+        (
+            # escaped text in middle of prompt with equal assignment
+            "this is a prompt {start}--bar=1.0{end} --baz=2.0",
+            "this is a prompt {start}--bar=1.0{end}",
+            {"baz": "2.0"},
+        ),
+        (
+            # escaped text in middle of prompt with space assignment
+            "this is a prompt {start}--bar 1.0{end} --baz 2.0",
+            "this is a prompt {start}--bar 1.0{end}",
+            {"baz": "2.0"},
+        ),
+        (
+            # escaped text in multiple escape sequences
+            'this is a prompt --bar=1.0 {start}my first context block{end} and then ```my second context block``` --baz=2.0',
+            'this is a prompt  {start}my first context block{end} and then ```my second context block```',
+            {'bar': '1.0', 'baz': '2.0'},
+        ),
+        (
+            # entire prompt is escaped
+            "{start}this is a prompt --bar=1.0 --baz=2.0{end}",
+            "{start}this is a prompt --bar=1.0 --baz=2.0{end}",
+            {},
+        ),
+        (
+            # multi-line escaped text
+            "this is a prompt \n--bar=1.0 --baz=2.0\n{start}--foo=3.0 \n another line \nmy final line{end}",
+            "this is a prompt \n \n{start}--foo=3.0 \n another line \nmy final line{end}",
+            {'bar': '1.0', 'baz': '2.0'},
+        )
+
+    ]
+
+    delimiters = ["```", '"""', "`"]
+
+    for start, end in [(d, d) for d in delimiters]:
+        for prompt, expected_prompt, expected_args in test_cases:
+            formatted_prompt = prompt.format(start=start, end=end)
+            formatted_expected_prompt = expected_prompt.format(start=start, end=end)
+            assert parse_args(formatted_prompt) == (
+                formatted_expected_prompt,
+                expected_args,
+            )
\ No newline at end of file