Repository: hylang/hy

Issue:
Bug in reader: can't terminate string literals containing \ at end of line.
Common Lisp had docstrings before Python existed. Hy should totally support this.

``` Python
>>> def foo():
    """I'm a docstring!"""
    return 42

>>> help(foo)
Help on function foo in module __main__:

foo()
    I'm a docstring!

>>> foo.__doc__
"I'm a docstring!"
```

Let's try it in Hy.

```
=> (defn foo ()
... """I'm a docstring"""
... 42)
=> (help foo)
Help on function foo:

foo()

=> foo.__doc__
''
```

?!
Where's my docstring?
(some time later...)
Let's try hy2py.

``` Python
def foo():
    ''
    "I'm a docstring!"
    ''
    return 42
```

I see what you did there, Hy. Yeah, there should probably be a warning about that somewhere. Triple  quotes don't work. Why does Python use them anyway? So we can include newlines mostly. Common Lisp strings do that already. Does Hy?

```
=> "one
... two"
'one\ntwo
```

Yup. Looks good.
Escapes also appear to work. But what if I actually wanted backslashes? Do raw strings work?

```
=> r"\foo"
'\\foo
```

Nice. Now **raw** triple quoted strings? This works in Python:

``` Python
>>> r"""\
foo"""
'\\\nfoo'
```

Let's use what we've learned and try it in Hy.

```
=> r"\
... foo"
... "
... ""
... """
... what?
... \
... \
... \\
... \\\
... \\\\
... \"
... \\"
... \\\"
... \\\\"
...   C-c C-c

KeyboardInterrupt
```

Something is broken...



Discussion:
> Common Lisp had docstrings before Python existed. Hy should totally support this [...]

As you've observed, Hy does not have Python-style triple-quoted string literals (it would be redundant, because newlines are allowed in ordinary string literals), but note that Hy _does_ support docstrings&mdash;

```
=> (defn foo []
...   "I am a docstring."
...   42)
=> (. foo __doc__)
'I am a docstring.'
```

This is [because](https://hg.python.org/cpython/file/b23ad0a6cf87/Lib/ast.py#l193) the Python runtime treats a string literal as a doctstring if it's the first statement in a function, class, or module definition.

I agree that your last example looks like a bug in the REPL (we expect one of the quotes to terminate the string literal, but it doesn't).

> note that Hy does support docstrings

Yes, I gathered from the hy2py output that the reason my docstring attempt failed was that the triple quoted string was instead interpreted as three strings and thus the docstring was actually empty. Sorry for not being clear. This is obvious to anyone who can read Python, but you couldn't assume I knew that. I shouldn't have assumed that everyone knew that I knew. Phew.

I was merely suggesting that there should be a note of this in the documentation, because it could confuse other Python programmers the same way.

> I agree that your last example looks like a bug in the REPL

Perhaps just in the reader? It doesn't have to be in interactive mode. I attempted to use hy2py on a file containing the offending string. It failed because the string wasn't terminated. Hy seems to consider this a syntax error, but I think we can agree this is not the expected behavior. This seems to happen anytime a newline immediately follows a backslash in any Hy string literal. As a workaround you could use a "\n" after the backslash, but, of course, this doesn't work in raw strings.

> I was merely suggesting that there should be a note of this in the documentation

On second thought, I think `hy --spy` and `disassemble` should be more prominent in the documentation instead. The docstring problem was immediately obvious once I saw the Python Hy generated.

> (it would be redundant, because newlines are allowed in ordinary string literals)

Newlines aren't the only reason to have triple quotes. It also lets you insert `'` and `"` without escapes:

``` Python
"""He said, "It doesn't work in Hy."."""
```

It isn't possible to avoid these escapes in Hy.

Python will also concatenate adjacent string literals:

``` Python
>>> "He " "said, " '"It ' "doesn't work in " 'Hy.".'
'He said, "it doesn\'t work in Hy.".'
>>> ('A line that is much too long to fit on one line, '
 "doesn't actually have to fit on one line")
"A line that is much too long to fit on one line, doesn't actually have to fit on one line"
```

Unlike `+` between strings, this happens at read time, not run time. Hy can't do this either.

I think Hy's string literal support could be better. It's not up to Python's level. Triple quotes are one option, but we could do even better by supporting arbitrary strings as terminators, like in Perl or C++. I've also seen this done in Common Lisp with reader macros, but Hy's reader macro's aren't powerful enough to do it.

But what's most concerning here is that I reported a bug in Hy months ago but it seems to have been completely forgotten.

> But what's most concerning here is that I reported a bug in Hy months ago but it seems to have been completely forgotten.

Well, I wouldn't say it's been _forgotten_ so much as that's it's been a victim of one of the little heartbreaking tragedies of our constrained and pitiful world, that even hackers as smart and enthusiastic as those reading the comments on this issue have but only a finite _and small_ amount of time and attention to allocate amongst the vastness of exciting and important things to hack on, that so many bug reports go unanswered, like prayers to an absent God.

@gilch, if no one else has yet been able to afford to bother, let me thank you for filing this issue; as far as my mortal brain can perceive, you are entirely correct that the Hy codebase is currently _wrong_ on this matter of string literals containing a backslash followed by a newline. (And, incidentally, _I_ was wrong to have prematurely spoken in my previous comment as if I had reason to believe the problem was in the REPL specifically rather than being a farther-reaching lexing problem, for no greater reason than having a [vague memory that someone had said something](https://github.com/hylang/hy/issues/305) about the REPL and multiline strings in the past.) For none of us should forget that it is _only_ by the grace of those who _care_ enough to file an issue (as you have filed), to write a patch (as [you have](https://github.com/hylang/hy/commits?author=gilch) written), to review a patch (as you have reviewed), that the people of our miserable excuse for a civilization get to enjoy the amount of existing and correct software that they do. 

But I don't have a fix for this one right now. Not because I don't care, but because&mdash;as an _empirical_ observation of fact, as shameful as it is to admit it out loud&mdash;I evidently don't care _enough_ (relative to everything else worth caring about) to have already fixed it.

I didn't mean to pick on you in particular, Zack. I'd be inclined to fix it myself if I understood Hy's internals better. Most of the other minor issues like this that I filed around the same time have already been dealt with.

I assumed that bugfixes like this were higher priority than new features. When that didn't happen I thought it was my fault for writing a meandering OP that took too long to get to the point. I shouldn't have included the surprising lack of triple quotes in the first place--it's distracting from the more important reader bugfix. It seemed that none but you, Zack, noticed my main point. I tried to correct my error (and, perhaps, get the attention of a third party better prepared to assist) with a more focused title (twice).

Perhaps I should just close this issue and replace it with a better-written one.

@gilch Well, the bug isn't _too_ high-priority, considering it only really affects the REPL.

Also, it's not your writing style. There were bugs that I opened a year ago that only got fixed recently. I actually kind of like the way you put the part that shows the REPL issue with raw strings. :)

> Also, it's not your writing style.

Thanks, Kirby, that makes me feel better.

It doesn't just affect the REPL though. Try putting a string like that in a `.hy` file and see what happens. That's why I thought it was in the reader (lexer?) and not just the REPL.

@gilch Huh..interesting...

I should take a look.


Resolving patch:
diff --git a/hy/lex/lexer.py b/hy/lex/lexer.py
--- a/hy/lex/lexer.py
+++ b/hy/lex/lexer.py
@@ -50,7 +50,7 @@
     "  # start string
     (?:
        | [^"\\]             # non-quote or backslash
-       | \\.                # or escaped single character
+       | \\(.|\n)           # or escaped single character or newline
        | \\x[0-9a-fA-F]{2}  # or escaped raw character
        | \\u[0-9a-fA-F]{4}  # or unicode escape
        | \\U[0-9a-fA-F]{8}  # or long unicode escape


Test patch:
diff --git a/tests/lex/test_lex.py b/tests/lex/test_lex.py
--- a/tests/lex/test_lex.py
+++ b/tests/lex/test_lex.py
@@ -98,8 +98,14 @@ def test_lex_symbols():
 
 def test_lex_strings():
     """ Make sure that strings are valid expressions"""
-    objs = tokenize("\"foo\" ")
+    objs = tokenize('"foo"')
     assert objs == [HyString("foo")]
+    # Make sure backslash-escaped newlines work (see issue #831)
+    objs = tokenize(r"""
+"a\
+bc"
+""")
+    assert objs == [HyString("abc")]
 
 
 def test_lex_integers():