Repository: evennia/evennia

Issue:
[BUG - Develop] funcparser breaks when args contain an odd number of apostrophe or quote marks
#### Describe the bug
When a string with an odd number of quote or apostrophe characters is put into a funcparser function, it fails to parse the function.

#### To Reproduce
Steps to reproduce the behavior:
1. Put `$crop(spider silk, 5)` through the parser and verify it works (output: `spide`)
2. Put `$crop(spider's silk, 5)` through the parser: one apostrophe, output: `$crop(spider's silk, 5)`
3. Then put `$crop(spider's' silk, 5)` through the parser: two apostrophies, output: `spide`
4. Lastly, put `$crop('spider's' silk, 5)` through the parser: three apostrophes, output: `$crop('spider's' silk, 5)`

#### Expected behavior
It should successfully parse the function when it's being given a string that contains a single quote mark or apostrophe.

#### Develop-branch commit
7f4769bd9


Discussion:
I see this as a limitation of how the parser is built.
It simulates what python would do and treats ' as the start of a literal string.
To avoid this, you can escape the ' so it is treated as a char.
Try
`$crop(spider\\'s silk, 5)`
It could also be improved by accepting something like "spider's silk", it does not seem to accept it currently
The funcparser is specifically built to support nesting of quotes; if it does not there may be a bug :) 
@Griatch problem here is the quote needs to be treated as a literal char, not as a quote
I can add a check for double-quote and single-quote bools, so you could do similar to python
`"spider's silk"` so everything inside is literal, this one currently doesn't work
Yes, it should work like Python in this case.
> I see this as a limitation of how the parser is built. It simulates what python would do and treats ' as the start of a literal string. To avoid this, you can escape the ' so it is treated as a char. Try `$crop(spider\\'s silk, 5)`

Unfortunately that's one of the first things we tried - it doesn't make a difference.
> > I see this as a limitation of how the parser is built. It simulates what python would do and treats ' as the start of a literal string. To avoid this, you can escape the ' so it is treated as a char. Try `$crop(spider\\'s silk, 5)`
> 
> Unfortunately that's one of the first things we tried - it doesn't make a difference.

weird, it works in my test case
Okay, my bad, escaping it DOES work.
Reference of discussion in discord:
 
It's a balance between being pythonic and easy to use; because plain python does require the quotes to denote a string while FuncParse really starts out as a string and should default to a string also if no quotes are given. 

So, 

- `$func(something)` is an implicit string `"something"`
- `$func("something")` is an explicit string `"something"`
- `$func(some'thing)` is an implicit string `"some'thing"`
- `$func("some'thing")` is an explicit string `"some'thing"`
- `$func('something')` is an explicit string `"something"` (etc, for '..."...' and so on)
- `$func(\"something\")` is an implicit string `"\"something\""` with visible quotes

That is, if the quotes are to be explicitly visible in the string, they need to be escaped, otherwise surrounding the argument with "..." or '....' is equivalent to leaving them out. 

(By the way, there is no concern of breaking backwards-compatibility here; unit tests can be updated to match behavior as needed. There is no FuncParser at all in master branch. Develop branch is not stable and people using a dev-branch accept the risks of using code that may still be in flux.)
Yeah, that breakdown is what I was thinking when I first posted this issue - or at least something similar. Since an unpaired quote mark isn't intended to be read by the parser as a syntactical element at all, it should be able to successfully parse the string.

That said: I think a big part of why this issue is cropping up is because of attempting to use both implicit and explicit string markers in the funcs that take strings, and it might be better to just not do that at all and just explicitly escape commas if needed? Especially since, at the moment, `$crop("spiders silk",5)` outputs `"spid`, which negates the whole using-quotes-to-mark-literals concept.

So:

* `$func(something)` receives the string `something`
* `$func("something")` receives the string `"something"`
* `$func(some'thing)` receives the string `some'thing`
* `$func(\"something\")` receives the string `"something"`

Then I believe the only things that would need to be explicitly escaped when using the funcparser are `,` and `=` since they're used for parsing args/kwargs
So, if I understand you right, you are suggesting modifying the parser to drop the 'literal mode' entered when starting a `"` or `'` entirely, and just passing whatever is in the argument list as verbatim strings (except commas/equal signs)? 

I did think of this alternative in the past. And you are right that it would resolve the present `$func(some'thing, 5)` issue. 

I'm however not sure it's worth to drop the convenience and clarity of being able to do `$func("There is a red, blue and green man there, smiling.", 5)` though, having to do `$func(There is a red\, blue and green man there\, smiling., 5)`. 

It would seem better to bite the sour apple and solve this special case with a look-ahead to see if a `'` is indeed the beginning of a  a verbatim string or if it's an apostrophe.

In that case, there's two issues that would need to be fixed, although the odd-count quotes is more pressing since it actually breaks the parser. The other one is that using quotes to mark string literals is, with the current funcparser, basically useless, because the quotes aren't stripped.

With the example list you gave above, the idea is that `$func(something)` and `$func("something")` should result in the arg `something`, while `$func(\"something\")` would result in the arg `"something"` with the visible quotes.

However, what actually happens is that `$func(something)` is the _only_ syntax that will result in the arg `something`. Both `$func("something")` and `$func(\"something\")` result in the arg of `"something"` - so if you don't want those enclosing quotes in your final string, you can't use them and you have to escape everything already, anyway. That's why I thought it might be more useful to just scrap the idea of using the quotes as literal string markers, since it doesn't really work at the moment anyway. (That is, it works in the sense of taking all the text inside the quotes as a string, but it doesn't work in the sense of actually being useful.)
Yeah, I understand what you mean, and I made the parser like that originally; but discussing and thinking about it now, I think it will be more rare to actually want to have verbatim quotes in the string than it is to want to have commas - and you should be able to 'escape' verbatim quotes they way you do it in Python, by alternating `"` and `'`, removing another need for a direct use of `\`. 

So yes, making the default be to not include the quotes would be a change of the funcparser behavior. The odd-quote case needs to be addressed, as well as properly 'nesting' double- and single quotes to escape each other (which I suspect the parser is not properly doing right now). 
As said, the feature is new to 1.0-dev and behavior is not stable, so changes like this can happen. Considering the intricacies of funcparser, it's suprising there haven't been more quirks with it so far :) 

(by the way, I don't expect you to work on this or anything; I can update the funcparser, it will still be a little while until I circle around to this though)
I'm working on it btw, did some refactoring to make the fix easier, should have it working soon.

Resolving patch:
diff --git a/evennia/utils/funcparser.py b/evennia/utils/funcparser.py
--- a/evennia/utils/funcparser.py
+++ b/evennia/utils/funcparser.py
@@ -84,8 +84,8 @@ class _ParsedFunc:
     # state storage
     fullstr: str = ""
     infuncstr: str = ""
-    single_quoted: bool = False
-    double_quoted: bool = False
+    single_quoted: int = -1
+    double_quoted: int = -1
     current_kwarg: str = ""
     open_lparens: int = 0
     open_lsquate: int = 0
@@ -318,8 +318,8 @@ def parse(
         # parsing state
         callstack = []
 
-        single_quoted = False
-        double_quoted = False
+        single_quoted = -1
+        double_quoted = -1
         open_lparens = 0  # open (
         open_lsquare = 0  # open [
         open_lcurly = 0  # open {
@@ -330,6 +330,7 @@ def parse(
         curr_func = None
         fullstr = ""  # final string
         infuncstr = ""  # string parts inside the current level of $funcdef (including $)
+        literal_infuncstr = False
 
         for char in string:
 
@@ -373,12 +374,13 @@ def parse(
                         curr_func.open_lcurly = open_lcurly
                         current_kwarg = ""
                         infuncstr = ""
-                        single_quoted = False
-                        double_quoted = False
+                        single_quoted = -1
+                        double_quoted = -1
                         open_lparens = 0
                         open_lsquare = 0
                         open_lcurly = 0
                         exec_return = ""
+                        literal_infuncstr = False
                         callstack.append(curr_func)
 
                 # start a new func
@@ -401,19 +403,41 @@ def parse(
                 infuncstr += str(exec_return)
                 exec_return = ""
 
-            if char == "'":  # note that this is the same as "\'"
+            if char == "'" and double_quoted < 0:  # note that this is the same as "\'"
                 # a single quote - flip status
-                single_quoted = not single_quoted
-                infuncstr += char
+                if single_quoted == 0:
+                    infuncstr = infuncstr[1:]
+                    single_quoted = -1
+                elif single_quoted > 0:
+                    prefix = infuncstr[0:single_quoted]
+                    infuncstr = prefix + infuncstr[single_quoted+1:]
+                    single_quoted = -1
+                else:
+                    infuncstr += char
+                    infuncstr = infuncstr.strip()
+                    single_quoted = len(infuncstr) - 1
+                    literal_infuncstr = True
+
                 continue
 
-            if char == '"':  # note that this is the same as '\"'
+            if char == '"' and single_quoted < 0:  # note that this is the same as '\"'
                 # a double quote = flip status
-                double_quoted = not double_quoted
-                infuncstr += char
+                if double_quoted == 0:
+                    infuncstr = infuncstr[1:]
+                    double_quoted = -1
+                elif double_quoted > 0:
+                    prefix = infuncstr[0:double_quoted]
+                    infuncstr = prefix + infuncstr[double_quoted + 1:]
+                    double_quoted = -1
+                else:
+                    infuncstr += char
+                    infuncstr = infuncstr.strip()
+                    double_quoted = len(infuncstr) - 1
+                    literal_infuncstr = True
+
                 continue
 
-            if double_quoted or single_quoted:
+            if double_quoted >= 0 or single_quoted >= 0:
                 # inside a string definition - this escapes everything else
                 infuncstr += char
                 continue
@@ -477,12 +501,15 @@ def parse(
                     else:
                         curr_func.args.append(exec_return)
                 else:
+                    if not literal_infuncstr:
+                        infuncstr = infuncstr.strip()
+
                     # store a string instead
                     if current_kwarg:
-                        curr_func.kwargs[current_kwarg] = infuncstr.strip()
-                    elif infuncstr.strip():
+                        curr_func.kwargs[current_kwarg] = infuncstr
+                    elif literal_infuncstr or infuncstr.strip():
                         # don't store the empty string
-                        curr_func.args.append(infuncstr.strip())
+                        curr_func.args.append(infuncstr)
 
                 # note that at this point either exec_return or infuncstr will
                 # be empty. We need to store the full string so we can print
@@ -493,6 +520,7 @@ def parse(
                 current_kwarg = ""
                 exec_return = ""
                 infuncstr = ""
+                literal_infuncstr = False
 
                 if char == ")":
                     # closing the function list - this means we have a
@@ -536,6 +564,7 @@ def parse(
                         if return_str:
                             exec_return = ""
                         infuncstr = ""
+                        literal_infuncstr = False
                 continue
 
             infuncstr += char


Test patch:
diff --git a/evennia/utils/tests/test_funcparser.py b/evennia/utils/tests/test_funcparser.py
--- a/evennia/utils/tests/test_funcparser.py
+++ b/evennia/utils/tests/test_funcparser.py
@@ -44,6 +44,7 @@ def _double_callable(*args, **kwargs):
 def _eval_callable(*args, **kwargs):
     if args:
         return simple_eval(args[0])
+
     return ""
 
 
@@ -113,25 +114,25 @@ def setUp(self):
             ("$foo() Test noargs5", "_test() Test noargs5"),
             ("Test args1 $foo(a,b,c)", "Test args1 _test(a, b, c)"),
             ("Test args2 $bar(foo, bar,    too)", "Test args2 _test(foo, bar, too)"),
-            ("Test args3 $bar(foo, bar, '   too')", "Test args3 _test(foo, bar, '   too')"),
-            ("Test args4 $foo('')", "Test args4 _test('')"),
-            ('Test args4 $foo("")', 'Test args4 _test("")'),
+            (r"Test args3 $bar(foo, bar, '   too')", "Test args3 _test(foo, bar,    too)"),
+            ("Test args4 $foo('')", "Test args4 _test()"),
+            ('Test args4 $foo("")', 'Test args4 _test()'),
             ("Test args5 $foo(\(\))", "Test args5 _test(())"),
             ("Test args6 $foo(\()", "Test args6 _test(()"),
             ("Test args7 $foo(())", "Test args7 _test(())"),
             ("Test args8 $foo())", "Test args8 _test())"),
             ("Test args9 $foo(=)", "Test args9 _test(=)"),
             ("Test args10 $foo(\,)", "Test args10 _test(,)"),
-            ("Test args10 $foo(',')", "Test args10 _test(',')"),
+            ("Test args10 $foo(',')", "Test args10 _test(,)"),
             ("Test args11 $foo(()", "Test args11 $foo(()"),  # invalid syntax
             (
                 "Test kwarg1 $bar(foo=1, bar='foo', too=ere)",
-                "Test kwarg1 _test(foo=1, bar='foo', too=ere)",
+                "Test kwarg1 _test(foo=1, bar=foo, too=ere)",
             ),
             ("Test kwarg2 $bar(foo,bar,too=ere)", "Test kwarg2 _test(foo, bar, too=ere)"),
             ("test kwarg3 $foo(foo = bar, bar = ere )", "test kwarg3 _test(foo=bar, bar=ere)"),
             (
-                "test kwarg4 $foo(foo =' bar ',\" bar \"= ere )",
+                r"test kwarg4 $foo(foo =\' bar \',\" bar \"= ere )",
                 "test kwarg4 _test(foo=' bar ', \" bar \"=ere)",
             ),
             (
@@ -180,22 +181,24 @@ def setUp(self):
             ("Test clr $clr(r, This is a red string!)", "Test clr |rThis is a red string!|n"),
             ("Test eval1 $eval(21 + 21 - 10)", "Test eval1 32"),
             ("Test eval2 $eval((21 + 21) / 2)", "Test eval2 21.0"),
-            ("Test eval3 $eval('21' + 'foo' + 'bar')", "Test eval3 21foobar"),
-            ("Test eval4 $eval('21' + '$repl()' + '' + str(10 // 2))", "Test eval4 21rr5"),
-            ("Test eval5 $eval('21' + '\$repl()' + '' + str(10 // 2))", "Test eval5 21$repl()5"),
-            ("Test eval6 $eval('$repl(a)' + '$repl(b)')", "Test eval6 rarrbr"),
+            ("Test eval3 $eval(\"'21' + 'foo' + 'bar'\")", "Test eval3 21foobar"),
+            (r"Test eval4 $eval(\'21\' + \'$repl()\' + \"''\" + str(10 // 2))", "Test eval4 21rr5"),
+            (r"Test eval5 $eval(\'21\' + \'\$repl()\' + \'\' + str(10 // 2))", "Test eval5 21$repl()5"),
+            ("Test eval6 $eval(\"'$repl(a)' + '$repl(b)'\")", "Test eval6 rarrbr"),
             ("Test type1 $typ([1,2,3,4])", "Test type1 <class 'list'>"),
             ("Test type2 $typ((1,2,3,4))", "Test type2 <class 'tuple'>"),
             ("Test type3 $typ({1,2,3,4})", "Test type3 <class 'set'>"),
             ("Test type4 $typ({1:2,3:4})", "Test type4 <class 'dict'>"),
             ("Test type5 $typ(1), $typ(1.0)", "Test type5 <class 'int'>, <class 'float'>"),
-            ("Test type6 $typ('1'), $typ(\"1.0\")", "Test type6 <class 'str'>, <class 'str'>"),
+            ("Test type6 $typ(\"'1'\"), $typ('\"1.0\"')", "Test type6 <class 'str'>, <class 'str'>"),
             ("Test add1 $add(1, 2)", "Test add1 3"),
             ("Test add2 $add([1,2,3,4], [5,6])", "Test add2 [1, 2, 3, 4, 5, 6]"),
             ("Test literal1 $sum($lit([1,2,3,4,5,6]))", "Test literal1 21"),
             ("Test literal2 $typ($lit(1))", "Test literal2 <class 'int'>"),
             ("Test literal3 $typ($lit(1)aaa)", "Test literal3 <class 'str'>"),
             ("Test literal4 $typ(aaa$lit(1))", "Test literal4 <class 'str'>"),
+            ("Test spider's thread", "Test spider's thread"),
+
         ]
     )
     def test_parse(self, string, expected):
@@ -258,7 +261,11 @@ def test_parse_lit(self):
         self.assertEqual([1, 2, 3, 4], ret)
         self.assertTrue(isinstance(ret, list))
 
-        ret = self.parser.parse_to_any("$lit('')")
+        ret = self.parser.parse_to_any("$lit(\"''\")")
+        self.assertEqual("", ret)
+        self.assertTrue(isinstance(ret, str))
+
+        ret = self.parser.parse_to_any(r"$lit(\'\')")
         self.assertEqual("", ret)
         self.assertTrue(isinstance(ret, str))
 
@@ -390,7 +397,8 @@ def test_conjugate(self, string, expected_you, expected_them):
             ("Some $rjust(Hello, 30)", "Some                          Hello"),
             ("Some $rjust(Hello, width=30)", "Some                          Hello"),
             ("Some $cjust(Hello, 30)", "Some             Hello             "),
-            ("Some $eval('-'*20)Hello", "Some --------------------Hello"),
+            ("Some $eval(\"'-'*20\")Hello", "Some --------------------Hello"),
+            ("$crop(\"spider's silk\", 5)", "spide"),
         ]
     )
     def test_other_callables(self, string, expected):
@@ -455,18 +463,18 @@ def test_escaped(self):
             self.parser.parse(
                 "this should be $pad('''escaped,''' and '''instead,''' cropped $crop(with a long,5) text., 80)"
             ),
-            "this should be              '''escaped,''' and '''instead,''' cropped with  text.              ",
+            "this should be                    escaped, and instead, cropped with  text.                    ",
         )
 
     def test_escaped2(self):
+        raw_str = 'this should be $pad("""escaped,""" and """instead,""" cropped $crop(with a long,5) text., 80)'
+        expected = 'this should be                    escaped, and instead, cropped with  text.                    '
+        result = self.parser.parse(raw_str)
         self.assertEqual(
-            self.parser.parse(
-                'this should be $pad("""escaped,""" and """instead,""" cropped $crop(with a long,5) text., 80)'
-            ),
-            'this should be              """escaped,""" and """instead,""" cropped with  text.              ',
+            result,
+            expected,
         )
 
-
 class TestCallableSearch(test_resources.BaseEvenniaTest):
     """
     Test the $search(query) callable