Repository: hylang/hy

Issue:
Add support for "=" syntax in  f-strings
Since 3.8, Python has had "[equal sign syntax](https://docs.python.org/3/reference/lexical_analysis.html#formatted-string-literals)" in f-strings, which can be useful for debugging. For example:

```
In [2]: the_string = "foo"

In [9]: f"{the_string=}"
Out[12]: "the_string='foo'"

In [13]: f"{the_string=:5}"
Out[16]: 'the_string=foo  '

In [19]: from datetime import datetime; the_object=datetime.now()

In [29]: f"{the_object=!r}"
Out[32]: 'the_object=datetime.datetime(2021, 1, 24, 21, 12, 23, 550361)'

In [33]: f"{the_object=!s}"
Out[33]: 'the_object=2021-01-24 21:12:23.550361'

In [34]:
```

It would be great to have this in Hy. It looks like this could be handled in `_format_string` in `hy.compiler`, after the handling for format specs. The simple case (`f"{x=}"`) could potentially be handled in place, with the case of a trailing format specification being handled recursively.

Would you consider adding this in spite of it being for Python >=3.8? This is cool - since the parsing is handled by Hy, I think we could make this feature compatible for Hy running on older versions of Python. An example (excuse the clumsy use of pickle to ship the tree around different interpreters, I'm new to `ast`):

```
Python 3.9.1 (default, Dec 13 2020, 11:55:53)

In [97]: tree = ast.parse("print(f'{foo=}')")

In [121]: pickle.dump(tree, open("/home/joshua/test.pickle", "wb"))
```

```
Python 3.6.12 (default, Jan 24 2021, 16:44:51)

>>> tree = pickle.load(open("/home/joshua/test.pickle", "rb"))
>>> tree
<_ast.Module object at 0x7fae44ac09e8>
>>> compile(tree, filename="string", mode="exec")
<code object <module> at 0x7fae44981270, file "string", line 1>
>>> eval(compile(tree, filename="string", mode="exec"))
foo='bar'
```

I'm happy to work on this if it's desirable, and happy for any pointers also.


Discussion:
I'd love to have this, and you're more than welcome to take a crack at it. How do you imagine the syntax working, exactly?

> I think we could make this feature compatible for Hy running on older versions of Python

It's not mandatory, but it's always nice to have backports like that, yeah. Hy supported f-strings themselves on earlier CPythons than Python itself did. The only reason that code is gone now is that those CPythons hit end-of-life (#1931).
The reference for the current syntax is https://docs.hylang.org/en/stable/language/syntax.html#syntax-fstrings .
In terms of syntax, we can work with Hy's existing handling of the Python [grammar](https://docs.python.org/3/reference/lexical_analysis.html#formatted-string-literals):

`replacement_field ::=  "{" f_expression ["="] ["!" conversion] [":" format_spec] "}"`

Current behaviour requires that the optional conversion and/or format_spec are delimited from a symbol by whitespace, and I think that's sane:

```
=> (setv string= =)
=> f"{string=!r}"
Traceback (most recent call last):
  File "stdin-1678ffe848442964fc78536004b8ec063332bbf2", line 1, in <module>
    f"{string=!r}"
NameError: name 'hyx_stringXequals_signXXexclamation_markXr' is not defined
=> f"{string= !r}"
'<function hyx_Xequals_signX at 0x7f3dd5152b80>'
```

Naturally not an issue with parens:

```
=> f"{(do string=)!r:_<60}"
'<function hyx_Xequals_signX at 0x7f3dd5152b80>______________'
```

If we allow a "=" before conversion/format_spec, I can see it being error prone with particular symbols and/or complex format specs:

```
=> f"{string= =}"
=> f"{string= =!r:_<60}"
```

but Hy will throw a name error on something like `f"{string==!r}"` (except in the case that `string==!r` has been defined) so I don't think it's an issue.

I'll wait to see what happens with #1950 before going any further with this.
I think you will in some cases also need a space before the `=`, in case the expression is a symbol whose name ends with `=`.
Ah yes, I agree, perhaps that was unclear - there should be a space in the f-string before the `=` unless the `f_expression` has surrounding parentheses. Take a `string=` function as an example - `f"{string==}"` will result in trying to look up the symbol `string==`, whereas `f"{string= =}"` (or `f"{(do string=)=}"`) should print successfully.

Or did you mean in the output? That's a good point - a space after the `f_expression` and before the `=` could be useful in the case where the `f_expression` is a symbol ending in `=`, so the output is `"string= =<function string= ...>"` rather than `"string==<function string= ...>"`. It's worth considering the case of multiple substitions as well:

`f"{string= =}{string= =}"`

Using Python's existing formatting, this would print

`"string==<function string= ...>string==<function string= ...>"`

But, looking at it now I'm thinking it's not so bad, and this might be over-thinking. Introducing a space in the above would create confusing visual grouping - `"string= =<function string= ...>string= =<function string= ...>"`
I meant the input, not the output, yeah.
I have this more or less worked out, here's some examples of the behaviour:

```hy
;; '=' debugging syntax.
(setv foo "bar")
(assert (= f"{foo =}" "foo ='bar'"))

;; Whitespace is preserved.
(assert (= f"xyz{  foo = }" "xyz  foo = 'bar'"))

;; Explicit conversion is applied.
(assert (= f"{(do foo)=!s}" "(do foo)=bar"))

;; Format spec supercedes implicit conversion.
(setv pi 3.141593)
(assert (= f"{pi = :_^8.2f}" "pi = __3.14__"))

;; Format spec doesn't clobber the explicit conversion.
(with [(pytest.raises
	 ValueError
	 :match "Unknown format code 'f' for object of type 'str'")]
  f"{pi =!s:.3f}")
```

The handling of whitespace differs slightly from what we discussed above. On playing around and reading the spec<sup>[1](https://docs.python.org/3/reference/lexical_analysis.html#formatted-string-literals)</sup> more closely I found that in Python
> Spaces after the opening brace ‘'{'’, within the expression and after the ‘'='’ are all retained in the output.

To me it makes sense to emulate Python's behaviour as closely as possible, but it does mean we won't be able to get `"foo='bar'"` output from Hy, without providing some kind of delimiter.
That seems fine. Output with no whitespace is ugly and cramped anyway.

Resolving patch:
diff --git a/hy/lex/parser.py b/hy/lex/parser.py
--- a/hy/lex/parser.py
+++ b/hy/lex/parser.py
@@ -285,15 +285,24 @@ def _format_string(state, p, rest, allow_recursion=True):
        # Parse the first form.
        try:
            from . import parse_one_thing
-           model, item = parse_one_thing(item)
+           model, remainder = parse_one_thing(item)
+           f_expression = item[:-len(remainder)]
+           item = remainder
        except LexException:
            raise
        except ValueError as e:
            raise LexException.from_lexer("f-string: " + str(e), state, p[0])
        subnodes = [model]
 
+       # Check for '=' debugging syntax, reproduce whitespace in output
+       eq_sign_match = re.match(r'\s*=\s*', item)
+       if eq_sign_match:
+           values.append(HyString(f_expression + eq_sign_match.group()))
+           item = item[eq_sign_match.end():]
+       else:
+           item = item.lstrip()
+
        # Look for a conversion character.
-       item = item.lstrip()
        conversion = None
        if item.startswith('!'):
            conversion = item[1]
@@ -312,6 +321,9 @@ def _format_string(state, p, rest, allow_recursion=True):
            raise LexException.from_lexer(
                "f-string: trailing junk in field",
                state, p[0])
+       elif eq_sign_match and not conversion:
+           # Python has a special default conversion in this case.
+           conversion = "r"
 
        values.append(HyFComponent(subnodes, conversion=conversion))
 


Test patch:
diff --git a/tests/native_tests/language.hy b/tests/native_tests/language.hy
--- a/tests/native_tests/language.hy
+++ b/tests/native_tests/language.hy
@@ -1320,7 +1320,34 @@ cee\"} dee" "ey bee\ncee dee"))
   (with [(pytest.raises NameError)]
     (eval quoted))
   (setv world "goodbye")
-  (assert (= (eval quoted) "hello goodbye")))
+  (assert (= (eval quoted) "hello goodbye"))
+
+  ;; '=' debugging syntax.
+  (setv foo "bar")
+  (assert (= f"{foo =}" "foo ='bar'"))
+
+  ;; Whitespace is preserved.
+  (assert (= f"xyz{  foo = }" "xyz  foo = 'bar'"))
+
+  ;; Explicit conversion is applied.
+  (assert (= f"{ foo = !s}" " foo = bar"))
+
+  ;; Format spec supercedes implicit conversion.
+  (setv  pi 3.141593  fill "_")
+  (assert (= f"{pi = :{fill}^8.2f}" "pi = __3.14__"))
+
+  ;; Format spec doesn't clobber the explicit conversion.
+  (with [(pytest.raises
+           ValueError
+           :match r"Unknown format code '?f'? for object of type 'str'")]
+    f"{pi =!s:.3f}")
+
+  ;; Nested "=" is parsed, but fails at runtime, like Python.
+  (setv width 7)
+  (with [(pytest.raises
+           ValueError
+           :match r"I|invalid format spec(?:ifier)?")]
+    f"{pi =:{fill =}^{width =}.2f}"))
 
 
 (defn test-import-syntax []