Repository: hylang/hy

Issue:
Leading `-` should not get converted to underscores.
If you define a converter function with a name like
```
(defn ->foo [] ...)
```
Then it becomes module-private and you can't import it with `*`. (Unless you override this with `__all__`.) This is a gotcha that Hy doesn't need.

I propose we revise the mangling rules to distinguish leading `-`s from leading `_`s. Any internal `-` will continue to mangle to `_` as now, but if they're in the lead, it becomes `XhyphenHminusX`s, or preferably something shorter like `XsubX`s if we do the short names #1577. But leading `_` won't change.

We could also distinguish non-leading `_`s from `-`s for full reversibility by converting them to `Xlow_lineX` (or `XscoreX`).

Dunder names would look weird if we had to write them like `__init--`, so trailing `_`s will use the same rule as for the leading `_`s. Then we'd write `__init__` etc.

Examples,
```
=> (mangle "->>")
'hyx_XsubXXgtXgtX'
=> (mangle "->foo")
'hyx_XsubXXgtXfoo'
=> (mangle "foo->bar")
'hyx_foo_XgtXbar'
=> (mangle "_spam-eggs")
'_spam_eggs'
=> (mangle "_spam_eggs")
'_hyx_spamXscoreXeggs'
=> (mangle "__init__")
'__init__'
=> (mangle "--init--")
'hyx_XsubXXsubXinitXsubXXsubX'
=> (mangle "_-_")
'_hyx_XsubX_
=> (mangle "-_-")
'hyx_XsubXXscoreXXsubX'
=> (mangle "__")
'__'
```
In summary, special characters get mangled to X-quoted forms except for leading `_`s, trailing `_`s, and internal `-`, which all convert to `_`.


Discussion:
I have to admit that I never considered the possiblity of beginning a name with `->`; when I've seen that digraph in Lisp names before, it's been in the middle of the name.

> I propose we revise the mangling rules to distinguish leading `-`s from leading `_`s. Any internal `-` will continue to mangle to `_` as now, but if they're in the lead, it becomes `XhyphenHminusX`s, or preferably something shorter like `XsubX`s if we do the short names #1577. But leading `_` won't change.

That sounds reasonable.

> We could also distinguish non-leading `_`s from `-`s for full reversibility by converting them to `Xlow_lineX` (or `XscoreX`).

But in that case, the user is no longer allowed to write a Python name that contains an internal underscore, like `string.ascii_lowercase`, with a real underscore. The rule that any Python name is also a valid Hy name with the same meaning will be broken, and `mangle` will no longer be idempotent. The same goes for https://github.com/hylang/hy/issues/1634#issuecomment-396004072.
> The rule that any Python name is also a valid Hy name with the same meaning will be broken

An important point, which makes the leading underscore problem something of a separate question from this. The tradeoff is that we don't get full invertiblity. I'd prefer there be one-- and preferably only one --obvious way to do it. That is--always use hyphens internally. Always write the dunder names the same way, like `__init__`, not `--init--`. And while we're at it, always mangle `?` to `XqueryX` or something, instead of `is_`, since Python isn't consistent anyway.

> no longer be idempotent

I don't think idempotent functions can be invertible generally (except some trivial cases, like identity or singleton sets), since mapping the result back to itself when it started as something else means it can't be 1-to-1. We can't have both. So which property is more important? I thought it was invertiblity, since we're defining _and using_ an inverse function, `unmangle`. Why is idempotence better for this use case?
> So which property is more important? I thought it was invertiblity, since we're defining _and using_ an inverse function, `unmangle`.

`unmangle` isn't a real inverse, because `mangle` is many-to-one. Its purpose is to produce a pretty "Hy-like" name for the input, with "hyx_" prefixes gone and "XfooX" replaced with the character it indicates etc.

> Why is idempotence better for this use case?

When I think about it, "idempotence" in this case is just a fancy way to say the earlier thing I said, "any Python name is also a valid Hy name with the same meaning"; i.e., every Python identifier mangles to itself. This property is equivalent to idempotence because `mangle` is also guaranteed to always return a Python identifier.

So to answer your question, I would rather have Python names work unchanged as Hy names than have `unmangle` be a real inverse.
Welp, this is pretty much exactly what I proposed with #2005, I should have done my due diligence before creating a new issue.

+1 to `mangle` only affecting invalid python identifiers, I can go ahead and draft up a PR.
I went through and categorized all the issues and made a bit of an index for myself. I'll post it in discussions after i have the chance to clean it up some. 

Resolving patch:
diff --git a/hy/core/__init__.py b/hy/core/__init__.py
--- a/hy/core/__init__.py
+++ b/hy/core/__init__.py
@@ -1,8 +1,5 @@
 from .shadow import *
 from .language import *
 
-# Need to explicitly re-export names since some names (e.g. `-`)
-# start with a `_` after mangling
-from . import shadow
-from . import language
-__all__ = shadow.__all__ + language.__all__
+# Keep re-exports clean
+__all__ = [k for k in locals() if k[0] != '_' and k != 'hy']
diff --git a/hy/lex/__init__.py b/hy/lex/__init__.py
--- a/hy/lex/__init__.py
+++ b/hy/lex/__init__.py
@@ -98,13 +98,28 @@ def parse_one_thing(src_string):
 
 def mangle(s):
     """Stringify the argument and convert it to a valid Python identifier
-    according to Hy's mangling rules.
+    according to :ref:`Hy's mangling rules <mangling>`.
 
     Examples:
       ::
 
-         => (mangle "foo-bar")
-         'foo_bar'
+         => (mangle 'foo-bar)
+         "foo_bar"
+
+         => (mangle 'foo-bar?)
+         "is_foo_bar"
+
+         => (mangle '*)
+         "hyx_XasteriskX"
+
+         => (mangle '_foo/a?)
+         "_hyx_is_fooXsolidusXa"
+
+         => (mangle '-->)
+         "hyx_XhyphenHminusX_XgreaterHthan_signX"
+
+         => (mangle '<--)
+         "hyx_XlessHthan_signX__"
     """
     def unicode_char_to_hex(uchr):
         # Covert a unicode char to hex string, without prefix
@@ -119,13 +134,19 @@ def unicode_char_to_hex(uchr):
     if "." in s:
         return ".".join(mangle(x) if x else "" for x in s.split("."))
 
-    s = s.replace("-", "_")
+    # Step 1: Remove and save leading underscores
     s2 = s.lstrip('_')
     leading_underscores = '_' * (len(s) - len(s2))
     s = s2
 
+    # Step 2: Convert hyphens without introducing a new leading underscore
+    s = s[0] + s[1:].replace("-", "_") if s else s
+
+    # Step 3: Convert trailing `?` to leading `is_`
     if s.endswith("?"):
         s = 'is_' + s[:-1]
+
+    # Step 4: Convert invalid characters or reserved words
     if not isidentifier(leading_underscores + s):
         # Replace illegal characters with their Unicode character
         # names, or hexadecimal if they don't have one.
@@ -139,7 +160,9 @@ def unicode_char_to_hex(uchr):
                    or 'U{}'.format(unicode_char_to_hex(c)))
             for c in s)
 
+    # Step 5: Add back leading underscores
     s = leading_underscores + s
+
     assert isidentifier(s)
     return s
 
@@ -147,20 +170,40 @@ def unicode_char_to_hex(uchr):
 def unmangle(s):
     """Stringify the argument and try to convert it to a pretty unmangled
     form. This may not round-trip, because different Hy symbol names can
-    mangle to the same Python identifier.
+    mangle to the same Python identifier. See :ref:`Hy's mangling rules <mangling>`.
 
     Examples:
       ::
 
-         => (unmangle "foo_bar")
-         'foo-bar'
+         => (unmangle 'foo_bar)
+         "foo-bar"
+
+         => (unmangle 'is_foo_bar)
+         "foo-bar?"
+
+         => (unmangle 'hyx_XasteriskX)
+         "*"
+
+         => (unmangle '_hyx_is_fooXsolidusXa)
+         "_foo/a?"
+
+         => (unmangle 'hyx_XhyphenHminusX_XgreaterHthan_signX)
+         "-->"
+
+         => (unmangle 'hyx_XlessHthan_signX__)
+         "<--"
+
+         => (unmangle '__dunder_name__)
+         "__dunder-name__"
     """
 
     s = str(s)
 
-    s2 = s.lstrip('_')
-    leading_underscores = len(s) - len(s2)
-    s = s2
+    prefix = ""
+    suffix = ""
+    m = re.fullmatch(r'(_+)(.*?)(_*)', s, re.DOTALL)
+    if m:
+        prefix, s, suffix = m.groups()
 
     if s.startswith('hyx_'):
         s = re.sub('{0}(U)?([_a-z0-9H]+?){0}'.format(mangle_delim),
@@ -174,7 +217,7 @@ def unmangle(s):
         s = s[len("is_"):] + "?"
     s = s.replace('_', '-')
 
-    return '-' * leading_underscores + s
+    return prefix + s + suffix
 
 
 def read(from_file=sys.stdin, eof=""):


Test patch:
diff --git a/tests/native_tests/contrib/destructure.hy b/tests/native_tests/contrib/destructure.hy
--- a/tests/native_tests/contrib/destructure.hy
+++ b/tests/native_tests/contrib/destructure.hy
@@ -237,7 +237,7 @@
     "bar foo"
     [a b c d e f])
   (assert (= [1 2 3 4 5 6] (foo [1 2] {:c 3 :d 4} "e" 5 "f" 6)))
-  (assert (= "bar foo" foo.--doc--))
+  (assert (= "bar foo" foo.__doc__))
   (defn+ bar [&optional &rest &kwonly]
     (+ &optional &rest &kwonly))
   (assert (= (bar 1 2 3) 6)))
diff --git a/tests/native_tests/defclass.hy b/tests/native_tests/defclass.hy
--- a/tests/native_tests/defclass.hy
+++ b/tests/native_tests/defclass.hy
@@ -68,7 +68,7 @@
 (defn test-defclass-docstring []
   "NATIVE: test defclass docstring"
   (defclass A []
-    (setv --doc-- "doc string")
+    (setv __doc__ "doc string")
     (setv x 1))
   (setv a (A))
   (assert (= a.__doc__ "doc string"))
@@ -115,11 +115,11 @@
   (assert (.greet a) "hello"))
 
 (defn test-defclass-implicit-none-for-init []
-  "NATIVE: test that defclass adds an implicit None to --init--"
+  "NATIVE: test that defclass adds an implicit None to __init__"
   (defclass A []
-    (setv --init-- (fn [self] (setv self.x 1) 42)))
+    (setv __init__ (fn [self] (setv self.x 1) 42)))
   (defclass B []
-    (defn --init-- [self]
+    (defn __init__ [self]
       (setv self.x 2)
       42))
 
diff --git a/tests/native_tests/language.hy b/tests/native_tests/language.hy
--- a/tests/native_tests/language.hy
+++ b/tests/native_tests/language.hy
@@ -1625,11 +1625,11 @@ cee\"} dee" "ey bee\ncee dee"))
   (setv foo [(mycls) (mycls) (mycls)])
   (assert (is (. foo) foo))
   (assert (is (. foo [0]) (get foo 0)))
-  (assert (is (. foo [0] --class--) mycls))
-  (assert (is (. foo [1] --class--) mycls))
-  (assert (is (. foo [(+ 1 1)] --class--) mycls))
-  (assert (= (. foo [(+ 1 1)] --class-- --name-- [0]) "m"))
-  (assert (= (. foo [(+ 1 1)] --class-- --name-- [1]) "y"))
+  (assert (is (. foo [0] __class__) mycls))
+  (assert (is (. foo [1] __class__) mycls))
+  (assert (is (. foo [(+ 1 1)] __class__) mycls))
+  (assert (= (. foo [(+ 1 1)] __class__ __name__ [0]) "m"))
+  (assert (= (. foo [(+ 1 1)] __class__ __name__ [1]) "y"))
 
   (setv bar (mycls))
   (setv (. foo [1]) bar)
@@ -1826,20 +1826,20 @@ cee\"} dee" "ey bee\ncee dee"))
 
 (defn test-pep-3115 []
   (defclass member-table [dict]
-    (defn --init-- [self]
+    (defn __init__ [self]
       (setv self.member-names []))
 
-    (defn --setitem-- [self key value]
+    (defn __setitem__ [self key value]
       (if (not-in key self)
           (.append self.member-names key))
-      (dict.--setitem-- self key value)))
+      (dict.__setitem__ self key value)))
 
   (defclass OrderedClass [type]
-    (setv --prepare-- (classmethod (fn [metacls name bases]
+    (setv __prepare__ (classmethod (fn [metacls name bases]
       (member-table))))
 
-    (defn --new-- [cls name bases classdict]
-      (setv result (type.--new-- cls name bases (dict classdict)))
+    (defn __new__ [cls name bases classdict]
+      (setv result (type.__new__ cls name bases (dict classdict)))
       (setv result.member-names classdict.member-names)
       result))
 
@@ -1978,7 +1978,7 @@ cee\"} dee" "ey bee\ncee dee"))
 
 (defn test-pep-487 []
   (defclass QuestBase []
-    (defn --init-subclass-- [cls swallow #** kwargs]
+    (defn __init-subclass__ [cls swallow #** kwargs]
       (setv cls.swallow swallow)))
 
   (defclass Quest [QuestBase :swallow "african"])
diff --git a/tests/native_tests/mangling.hy b/tests/native_tests/mangling.hy
--- a/tests/native_tests/mangling.hy
+++ b/tests/native_tests/mangling.hy
@@ -13,8 +13,37 @@
   (assert (= -a__b- 2))
   (setv -_- 3)
   (assert (= -_- 3))
-  (assert (= --- 3))
-  (assert (= ___ 3)))
+  (assert (= -__ 3)))
+
+
+(defn test-mangle-hyphen-underscore []
+  ;; https://github.com/hylang/hy/issues/1635
+  (assert (= (mangle "-")               "hyx_XhyphenHminusX"))
+  (assert (= (mangle "-a")              "hyx_XhyphenHminusXa"))
+  (assert (= (mangle "-_a")             "hyx_XhyphenHminusX_a"))
+  (assert (= (mangle "_-a")             "_hyx_XhyphenHminusXa"))
+  (assert (= (mangle "__init__")        "__init__"))
+  (assert (= (mangle "--init--")        "hyx_XhyphenHminusX_init__"))
+  (assert (= (mangle "__dunder-name__") "__dunder_name__"))
+  (assert (= (mangle "-->")             "hyx_XhyphenHminusX_XgreaterHthan_signX"))
+  (assert (= (mangle "<--")             "hyx_XlessHthan_signX__"))
+
+  ;; test various interactions
+  (assert (= (mangle "----")   "hyx_XhyphenHminusX___"))
+  (assert (= (mangle "--__")   "hyx_XhyphenHminusX___"))
+  (assert (= (mangle "__--")   "__hyx_XhyphenHminusX_"))
+  (assert (= (mangle "__--__") "__hyx_XhyphenHminusX___"))
+  (assert (= (mangle "--?")    "hyx_is_XhyphenHminusX_"))
+  (assert (= (mangle "__--?")  "__hyx_is_XhyphenHminusX_"))
+
+  ;; test unmangling choices
+  (assert (= (unmangle "hyx_XhyphenHminusX")        "-"))
+  (assert (= (unmangle "_")                         "_"))
+  (assert (= (unmangle "__init__")                  "__init__"))
+  (assert (= (unmangle "hyx_XhyphenHminusX_init__") "--init--"))
+  (assert (= (unmangle "__dunder_name__")           "__dunder-name__"))
+  (assert (= (unmangle "hyx_XhyphenHminusX_XgreaterHthan_signX") "-->"))
+  (assert (= (unmangle "hyx_XlessHthan_signX__")                 "<--")))
 
 
 (defn test-underscore-number []
@@ -147,7 +176,7 @@
 
 
 (defn test-functions []
-  (for [[a b] [["---ab-cd?" "___is_ab_cd"]
+  (for [[a b] [["___ab-cd?" "___is_ab_cd"]
                ["if" "hyx_if"]
                ["⚘-⚘" "hyx_XflowerX_XflowerX"]]]
     (assert (= (mangle a) b))
diff --git a/tests/native_tests/native_macros.hy b/tests/native_tests/native_macros.hy
--- a/tests/native_tests/native_macros.hy
+++ b/tests/native_tests/native_macros.hy
@@ -327,9 +327,9 @@
 
 (defn test-defmain []
   "NATIVE: make sure defmain is clean"
-  (global --name--)
-  (setv oldname --name--)
-  (setv --name-- "__main__")
+  (global __name__)
+  (setv oldname __name__)
+  (setv __name__ "__main__")
 
   (defn main [x]
     (print (integer? x))
@@ -362,7 +362,7 @@
       (assert (= (str e) "1"))))
 
   (setv sys.argv oldargv)
-  (setv --name-- oldname))
+  (setv __name__ oldname))
 
 (defn test-macro-namespace-resolution []
   "Confirm that local versions of macro-macro dependencies do not shadow the
diff --git a/tests/native_tests/tag_macros.hy b/tests/native_tests/tag_macros.hy
--- a/tests/native_tests/tag_macros.hy
+++ b/tests/native_tests/tag_macros.hy
@@ -103,8 +103,8 @@
              (foo 1 2 3 :quux 4 :baz 5)))
 
   ;; @wraps preserved the docstring and __name__
-  (assert (= "foo" (. foo --name--)))
-  (assert (= "Bar." (. foo --doc--)))
+  (assert (= "foo" (. foo __name__)))
+  (assert (= "Bar." (. foo __doc__)))
 
   ;; We can use the #@ tag macro to apply more than one decorator
   #@(increment-arguments
diff --git a/tests/native_tests/with_test.hy b/tests/native_tests/with_test.hy
--- a/tests/native_tests/with_test.hy
+++ b/tests/native_tests/with_test.hy
@@ -3,14 +3,14 @@
 ;; license. See the LICENSE.
 
 (defclass WithTest [object]
-  (defn --init-- [self val]
+  (defn __init__ [self val]
     (setv self.val val)
     None)
 
-  (defn --enter-- [self]
+  (defn __enter__ [self]
     self.val)
 
-  (defn --exit-- [self type value traceback]
+  (defn __exit__ [self type value traceback]
     (setv self.val None)))
 
 (defn test-single-with []
@@ -45,8 +45,8 @@
         (assert (= t3 3))))
 
 (defclass SuppressZDE [object]
-  (defn --enter-- [self])
-  (defn --exit-- [self exc-type exc-value traceback]
+  (defn __enter__ [self])
+  (defn __exit__ [self exc-type exc-value traceback]
     (and (not (none? exc-type)) (issubclass exc-type ZeroDivisionError))))
 
 (defn test-exception-suppressing-with []
diff --git a/tests/resources/bin/__init__.hy b/tests/resources/bin/__init__.hy
--- a/tests/resources/bin/__init__.hy
+++ b/tests/resources/bin/__init__.hy
@@ -1,2 +1,2 @@
-(defn -null-fn-for-import-test []
+(defn _null-fn-for-import-test []
   pass)