Repository: hylang/hy

Issue:
`hy -c` doesn't clean up `sys.argv`
```
$ echo  '(import sys) (print sys.argv)' > args.hy
$ hy2py args.hy > args.py
```

In Python command line arguments are placed into `sys.argv` from index 1 (and index 0 is the name of the program).

```
$ python args.py a b c
['args.py', 'a', 'b', 'c']
```

Command line snippets adhere to above layout, with `-c` used as program name.

```
$ python -c "`cat args.py`" a b c
['-c', 'a', 'b', 'c']
```

Hy uses the same layout when a file is run.

```
$ hy args.hy a b c
['args.hy', 'a', 'b', 'c']
```

However, for a snippet it's something else.

```
$ hy -c "`cat args.hy`" a b c
['/home/csaba/.local/bin/hy', '-c', '(import sys) (print sys.argv)', 'a', 'b', 'c']
```
I need `(get sys.argv 3)` to access the first argument.

This is an unjustified departure from Python's ways and an unexpected inconsistency when switching between running program files and command line snippets.

For me personally, it partially beats the purpose of using Hy as a tool that bestows powers to run Python one-liners (I apologize if this purpose sounds lowly for a Lisp hacker).

```
$ hy --version
hy 0.18.0+18.gaa411bd
```



Discussion:
This is a bug. It looks like we have no tests for `hy -c` with further arguments.
I think making `-c` and `-m` interpret arguments as they ought to may require abandoning `argparse`. https://stackoverflow.com/questions/62364685
Randomly popping in: would it be possible to modify the argv array on -c to
add the -- and either continue or restart parsing from there or similar?

On Mon, Jun 15, 2020, 6:14 PM Kodi Arfer <notifications@github.com> wrote:

> I think making -c and -m interpret arguments as they ought to may require
> abandoning argparse. https://stackoverflow.com/questions/62364685
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/hylang/hy/issues/1894#issuecomment-644437870>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAM4YSKCTLX322PRDRNPHMTRW2TNXANCNFSM4NHUDLNA>
> .
>

I don't think `argparse` offers stream parsing. We would need to parse some options in order to figure out where to add the `--`, because `-c` might be the argument of another option, and by the time we've parsed the whole thing, it's too late.

Resolving patch:
diff --git a/hy/cmdline.py b/hy/cmdline.py
--- a/hy/cmdline.py
+++ b/hy/cmdline.py
@@ -485,42 +485,43 @@ def run_icommand(source, **kwargs):
     return run_repl(hr)
 
 
-USAGE = "%(prog)s [-h | -i cmd | -c cmd | -m module | file | -] [arg] ..."
-VERSION = "%(prog)s " + hy.__version__
+USAGE = "hy [-h | -v | -i CMD | -c CMD | -m MODULE | FILE | -] [ARG]..."
+VERSION = "hy " + hy.__version__
 EPILOG = """
-  file                  program read from script
-  module                module to execute as main
-  -                     program read from stdin
-  [arg] ...             arguments passed to program in sys.argv[1:]
+FILE
+  program read from script
+-
+  program read from stdin
+[ARG]...
+  arguments passed to program in sys.argv[1:]
 """
 
+class HyArgError(Exception): pass
 
 def cmdline_handler(scriptname, argv):
-    parser = argparse.ArgumentParser(
-        prog="hy",
-        usage=USAGE,
-        formatter_class=argparse.RawDescriptionHelpFormatter,
-        epilog=EPILOG)
-    parser.add_argument("-c", dest="command",
-                        help="program passed in as a string")
-    parser.add_argument("-m", dest="mod",
-                        help="module to run, passed in as a string")
-    parser.add_argument("-E", action='store_true',
-                        help="ignore PYTHON* environment variables")
-    parser.add_argument("-B", action='store_true',
-                        help="don't write .py[co] files on import; also PYTHONDONTWRITEBYTECODE=x")
-    parser.add_argument("-i", dest="icommand",
-                        help="program passed in as a string, then stay in REPL")
-    parser.add_argument("--spy", action="store_true",
-                        help="print equivalent Python code before executing")
-    parser.add_argument("--repl-output-fn",
-                        help="function for printing REPL output "
-                             "(e.g., hy.contrib.hy-repr.hy-repr)")
-    parser.add_argument("-v", "--version", action="version", version=VERSION)
-
-    # this will contain the script/program name and any arguments for it.
-    parser.add_argument('args', nargs=argparse.REMAINDER,
-                        help=argparse.SUPPRESS)
+    # We need to terminate interpretation of options after certain
+    # options, such as `-c`. So, we can't use `argparse`.
+
+    defs = [
+        dict(name=["-h", "--help"], action="help",
+            help="show this help message and exit"),
+        dict(name=["-c"], dest="command", terminate=True,
+            help="program passed in as a string"),
+        dict(name=["-m"], dest="mod", terminate=True,
+            help="module to run, passed in as a string"),
+        dict(name=["-E"], action='store_true',
+            help="ignore PYTHON* environment variables"),
+        dict(name=["-B"], action='store_true',
+            help="don't write .py[co] files on import; also PYTHONDONTWRITEBYTECODE=x"),
+        dict(name=["-i"], dest="icommand", terminate=True,
+            help="program passed in as a string, then stay in REPL"),
+        dict(name=["--spy"], action="store_true",
+            help="print equivalent Python code before executing"),
+        dict(name=["--repl-output-fn"], dest="repl_output_fn",
+            help="function for printing REPL output "
+                "(e.g., hy.contrib.hy-repr.hy-repr)"),
+        dict(name=["-v", "--version"], action="version",
+            help="show program's version number and exit")]
 
     # Get the path of the Hy cmdline executable and swap it with
     # `sys.executable` (saving the original, just in case).
@@ -531,47 +532,103 @@ def cmdline_handler(scriptname, argv):
     hy.sys_executable = sys.executable
     sys.executable = hy.executable
 
-    # Need to split the args.  If using "-m" all args after the MOD are sent to
-    # the module in sys.argv.
-    module_args = []
-    if "-m" in argv:
-        mloc = argv.index("-m")
-        if len(argv) > mloc+2:
-            module_args = argv[mloc+2:]
-            argv = argv[:mloc+2]
-
-    options = parser.parse_args(argv[1:])
+    program = argv[0]
+    argv = list(argv[1:])
+    options = {}
+
+    def err(fmt, *args):
+        raise HyArgError('hy: ' + fmt.format(*args))
+
+    def proc_opt(opt, arg=None, item=None, i=None):
+        matches = [o for o in defs if opt in o['name']]
+        if not matches:
+            err('unrecognized option: {}', opt)
+        [match] = matches
+        if 'dest' in match:
+            if arg:
+                pass
+            elif i is not None and i + 1 < len(item):
+                arg = item[i + 1 + (item[i + 1] == '='):]
+            elif argv:
+                arg = argv.pop(0)
+            else:
+                err('option {}: expected one argument', opt)
+            options[match['dest']] = arg
+        else:
+            options[match['name'][-1].lstrip('-')] = True
+        if 'terminate' in match:
+            return 'terminate'
+        return 'dest' in match
+
+    # Collect options.
+    while argv:
+        item = argv.pop(0)
+        if item == '--':
+            break
+        elif item.startswith('--'):
+            # One double-hyphen option.
+            opt, _, arg = item.partition('=')
+            if proc_opt(opt, arg=arg) == 'terminate':
+                break
+        elif item.startswith('-') and item != '-':
+            # One or more single-hyphen options.
+            for i in range(1, len(item)):
+                x = proc_opt('-' + item[i], item=item, i=i)
+                if x:
+                    break
+            if x == 'terminate':
+                break
+        else:
+            # We're done with options. Add the item back.
+            argv.insert(0, item)
+            break
 
-    if options.E:
-        # User did "hy -E ..."
+    if 'E' in options:
         _remove_python_envs()
 
-    if options.B:
+    if 'B' in options:
         sys.dont_write_bytecode = True
 
-    if options.command:
-        # User did "hy -c ..."
-        return run_command(options.command, filename='<string>')
+    if 'help' in options:
+        print('usage:', USAGE)
+        print('')
+        print('optional arguments:')
+        for o in defs:
+            print(', '.join(o['name']) +
+                ('=' + o['dest'].upper() if 'dest' in o else ''))
+            print('    ' + o['help'] +
+                (' (terminates option list)'
+                    if o.get('terminate')
+                    else ''))
+        print(EPILOG)
+        return 0
+
+    if 'version' in options:
+        print(VERSION)
+        return 0
+
+    if 'command' in options:
+        sys.argv = ['-c'] + argv
+        return run_command(options['command'], filename='<string>')
 
-    if options.mod:
-        # User did "hy -m ..."
-        sys.argv = [sys.argv[0]] + options.args + module_args
-        runpy.run_module(options.mod, run_name='__main__', alter_sys=True)
+    if 'mod' in options:
+        sys.argv = [program] + argv
+        runpy.run_module(options['mod'], run_name='__main__', alter_sys=True)
         return 0
 
-    if options.icommand:
-        # User did "hy -i ..."
-        return run_icommand(options.icommand, spy=options.spy,
-                            output_fn=options.repl_output_fn)
+    if 'icommand' in options:
+        return run_icommand(options['icommand'],
+            spy=options.get('spy'),
+            output_fn=options.get('repl_output_fn'))
 
-    if options.args:
-        if options.args[0] == "-":
+    if argv:
+        if argv[0] == "-":
             # Read the program from stdin
             return run_command(sys.stdin.read(), filename='<stdin>')
 
         else:
             # User did "hy <filename>"
-            filename = options.args[0]
+            filename = argv[0]
 
             # Emulate Python cmdline behavior by setting `sys.path` relative
             # to the executed file's location.
@@ -581,7 +638,7 @@ def cmdline_handler(scriptname, argv):
                 sys.path.insert(0, os.path.split(filename)[0])
 
             try:
-                sys.argv = options.args
+                sys.argv = argv
                 with filtered_hy_exceptions():
                     runhy.run_path(filename, run_name='__main__')
                 return 0
@@ -593,14 +650,19 @@ def cmdline_handler(scriptname, argv):
                 hy_exc_handler(*sys.exc_info())
                 sys.exit(1)
 
-    # User did NOTHING!
-    return run_repl(spy=options.spy, output_fn=options.repl_output_fn)
+    return run_repl(
+        spy=options.get('spy'),
+        output_fn=options.get('repl_output_fn'))
 
 
 # entry point for cmd line script "hy"
 def hy_main():
     sys.path.insert(0, "")
-    sys.exit(cmdline_handler("hy", sys.argv))
+    try:
+        sys.exit(cmdline_handler("hy", sys.argv))
+    except HyArgError as e:
+        print(e)
+        exit(1)
 
 
 def hyc_main():


Test patch:
diff --git a/tests/native_tests/language.hy b/tests/native_tests/language.hy
--- a/tests/native_tests/language.hy
+++ b/tests/native_tests/language.hy
@@ -1587,7 +1587,7 @@ cee\"} dee" "ey bee\ncee dee"))
 leaky()
 macros()
 "))
-  (assert (= (re.sub r"[L() ]" "" (disassemble `(+ ~(+ 1 1) 40) True))
+  (assert (= (re.sub r"[() ]" "" (disassemble `(+ ~(+ 1 1) 40) True))
              "2+40\n")))
 
 
diff --git a/tests/resources/bin/main.hy b/tests/resources/bin/main.hy
--- a/tests/resources/bin/main.hy
+++ b/tests/resources/bin/main.hy
@@ -1,5 +1,5 @@
 (defmain [&rest args]
-  (print args)
+  (print (+ "<" (.join "|" (cut args 1)) ">"))
   (print "Hello World")
   (if (in "exit1" args)
     1))
diff --git a/tests/test_bin.py b/tests/test_bin.py
--- a/tests/test_bin.py
+++ b/tests/test_bin.py
@@ -228,18 +228,18 @@ def test_bin_hy_stdin_bad_repr():
 
 def test_bin_hy_stdin_hy_repr():
     output, _ = run_cmd("hy", '(+ [1] [2])')
-    assert "[1, 2]" in output.replace('L', '')
+    assert "[1, 2]" in output
 
     output, _ = run_cmd(hr(), '(+ [1] [2])')
     assert "[1 2]" in output
 
     output, _ = run_cmd(hr("--spy"), '(+ [1] [2])')
-    assert "[1]+[2]" in output.replace('L', '').replace(' ', '')
+    assert "[1]+[2]" in output.replace(' ', '')
     assert "[1 2]" in output
 
     # --spy should work even when an exception is thrown
     output, _ = run_cmd(hr("--spy"), '(+ [1] [2] (foof))')
-    assert "[1]+[2]" in output.replace('L', '').replace(' ', '')
+    assert "[1]+[2]" in output.replace(' ', '')
 
 def test_bin_hy_ignore_python_env():
     os.environ.update({"PYTHONTEST": '0'})
@@ -271,6 +271,12 @@ def test_bin_hy_cmd():
         """hy -c '(setv x "bing") (defn f [] (+ "fiz" x)) (print (f))'""")
     assert 'fizbing' in output
 
+    # https://github.com/hylang/hy/issues/1894
+    output, _ = run_cmd(' '.join(('hy -c ',
+        repr('(import sys) (print (+ "<" (.join "|" sys.argv) ">"))'),
+        'AA', 'ZZ', '-m')))
+    assert '<-c|AA|ZZ|-m>' in output
+
 def test_bin_hy_icmd():
     output, _ = run_cmd("hy -i \"(koan)\"", "(ideas)")
     assert "monk" in output
@@ -287,7 +293,7 @@ def test_bin_hy_icmd_file():
     assert file_relative_path in output
 
 def test_bin_hy_icmd_and_spy():
-    output, _ = run_cmd("hy -i \"(+ [] [])\" --spy", "(+ 1 1)")
+    output, _ = run_cmd("hy --spy -i \"(+ [] [])\"", "(+ 1 1)")
     assert "[] + []" in output
 
 
@@ -343,9 +349,8 @@ def test_bin_hy_main():
 
 
 def test_bin_hy_main_args():
-    output, _ = run_cmd("hy tests/resources/bin/main.hy test 123")
-    assert "test" in output
-    assert "123" in output
+    output, _ = run_cmd("hy tests/resources/bin/main.hy test 123 -m -B 9")
+    assert "<test|123|-m|-B|9>" in output
 
 
 def test_bin_hy_main_exitvalue():
@@ -432,9 +437,8 @@ def test_bin_hy_file_sys_path():
 
 
 def test_bin_hy_module_main_args():
-    output, _ = run_cmd("hy -m tests.resources.bin.main test 123")
-    assert "test" in output
-    assert "123" in output
+    output, _ = run_cmd("hy -m tests.resources.bin.main test 123 -B")
+    assert "<test|123|-B>" in output
 
 
 def test_bin_hy_module_main_exitvalue():