Repository: pex-tool/pex

Issue:
Execute pex archive at runtime with -m parameter
We would like to create a pex without a fixed entry point and provide different entry points at runtime.
The doc indicates that this can be achieved  by tuning the environment variable  PEX_MODULE

> PEX_MODULE
Override the entry point into the PEX file. Can either be a module, e.g. ‘SimpleHTTPServer’, or a specific entry point in module:symbol form, e.g. “myapp.bin:main”.
>
https://pex.readthedocs.io/en/stable/api/index.html#module-pex.variables

This indeed works but is a bit counter intuitive. 
It would be nice to provide the entry point with the -m argument similar to how the python interpreter works but this doesn't work
`./mypexfile.pex -m mymodule arg0 arg1 arg2`

Is this something that can be implemented or it there a technical restriction not to to this ? In theory as we already propagate the arguments in the __main__ file we could also remove the beginning and redirect to the right module ?




Discussion:
Providing a module as entry point that does the redirection (like runpy) actually works. Nevertheless shouldn't this be something implemented by default ?
```
pex -m runpy -o interpreter.pex --python=python3.6
./interpreter.pex http.server
```
@fhoering maybe - it's slightly tricky. So clearly, if there is no entrypoint defined for the pex at build time, having pex look for `-m` at runtime steps on no ones toes. If an entrypoint is defined at build time though, pex should probably not stomp on `-m`, since the builtin entrypoint may expose a `-m` argument which should pass through unused by pex to the entrypoint. If you agree, then we have a perhaps strange  split in pex runtime behvior - namely `-m` works sometimes, at other times you need to `PEX_MODULE`. Today you are assured `PEX_MODULE` always works so there is no ambiguity, with the downside the built pex doesn't act exactly like a python interpreter.

Do you have any idea for principles we could abide by to resolve this tension?
I agree that having it parsed all the time and therefore users can't use the python interpreter options (-m -c, ..) anymore seems too intrusive.

Having it only set when no entry point is specified seems acceptable to me. When no entry point is specified it behaves like the python interpreter. Easy to explain as well.
Another option would be to provide some default template and activate it when invoking the pex command line tool with a special option.
Still another option would be to use the linux double dash syntax (--) to separate the arguments. 

My use case is that I want to use pex to ship code to the pyspark executors.  I started fixing pyspark code (https://github.com/fhoering/spark/commit/23447d52bebf608dcf044703c6df96d9a8b1d5dc (pythonexec can be provided, so it would be mypackpackage.pex) but finally I think this should be included in the pex package. So I will provide a custom entry point using runpy that redirects to the according module. 
I still think it should be possible by default or at least documented somehow. I personally prefer the double dash syntax but this won't work for my use case on pyspark.





Resolving patch:
diff --git a/pex/pex.py b/pex/pex.py
--- a/pex/pex.py
+++ b/pex/pex.py
@@ -431,18 +431,29 @@ def demote_bootstrap(cls):
     TRACER.log('  * - paths that do not exist or will be imported via zipimport')
 
   def execute_interpreter(self):
-    if sys.argv[1:]:
-      program = sys.argv[1]
-      try:
-        if program == '-':
-          content = sys.stdin.read()
-        else:
-          with open(program) as fp:
-            content = fp.read()
-      except IOError as e:
-        die("Could not open %s in the environment [%s]: %s" % (program, sys.argv[0], e))
-      sys.argv = sys.argv[1:]
-      self.execute_content(program, content)
+    args = sys.argv[1:]
+    if args:
+      # NB: We take care here to setup sys.argv to match how CPython does it for each case.
+      arg = args[0]
+      if arg == '-c':
+        content = args[1]
+        sys.argv = ['-c'] + args[2:]
+        self.execute_content('-c <cmd>', content, argv0='-c')
+      elif arg == '-m':
+        module = args[1]
+        sys.argv = args[1:]
+        self.execute_module(module)
+      else:
+        try:
+          if arg == '-':
+            content = sys.stdin.read()
+          else:
+            with open(arg) as fp:
+              content = fp.read()
+        except IOError as e:
+          die("Could not open %s in the environment [%s]: %s" % (arg, sys.argv[0], e))
+        sys.argv = args
+        self.execute_content(arg, content)
     else:
       self.demote_bootstrap()
 


Test patch:
diff --git a/tests/test_pex.py b/tests/test_pex.py
--- a/tests/test_pex.py
+++ b/tests/test_pex.py
@@ -347,6 +347,40 @@ def test_activate_interpreter_different_from_current():
           pytest.fail('PEX activation of %s failed with %s' % (pex, e))
 
 
+def test_execute_interpreter_dashc_program():
+  with temporary_dir() as pex_chroot:
+    pex_builder = PEXBuilder(path=pex_chroot)
+    pex_builder.freeze()
+    process = PEX(pex_chroot).run(args=['-c', 'import sys; print(" ".join(sys.argv))', 'one'],
+                                  stdout=subprocess.PIPE,
+                                  stderr=subprocess.PIPE,
+                                  blocking=False)
+    stdout, stderr = process.communicate()
+
+    assert 0 == process.returncode
+    assert b'-c one\n' == stdout
+    assert b'' == stderr
+
+
+def test_execute_interpreter_dashm_module():
+  with temporary_dir() as pex_chroot:
+    pex_builder = PEXBuilder(path=pex_chroot)
+    with temporary_file(root_dir=pex_chroot) as fp:
+      fp.write(b'import sys; print(" ".join(sys.argv))')
+      fp.close()
+      pex_builder.add_source(fp.name, 'foo/bar.py')
+    pex_builder.freeze()
+    process = PEX(pex_chroot).run(args=['-m', 'foo.bar', 'one', 'two'],
+                                  stdout=subprocess.PIPE,
+                                  stderr=subprocess.PIPE,
+                                  blocking=False)
+    stdout, stderr = process.communicate()
+
+    assert 0 == process.returncode
+    assert b'foo.bar one two\n' == stdout
+    assert b'' == stderr
+
+
 def test_execute_interpreter_stdin_program():
   with temporary_dir() as pex_chroot:
     pex_builder = PEXBuilder(path=pex_chroot)
@@ -356,13 +390,31 @@ def test_execute_interpreter_stdin_program():
                                   stderr=subprocess.PIPE,
                                   stdin=subprocess.PIPE,
                                   blocking=False)
-    stdout, stderr = process.communicate(input=b'import sys; print(" ".join(sys.argv[1:]))')
+    stdout, stderr = process.communicate(input=b'import sys; print(" ".join(sys.argv))')
 
     assert 0 == process.returncode
-    assert b'one two\n' == stdout
+    assert b'- one two\n' == stdout
     assert b'' == stderr
 
 
+def test_execute_interpreter_file_program():
+  with temporary_dir() as pex_chroot:
+    pex_builder = PEXBuilder(path=pex_chroot)
+    pex_builder.freeze()
+    with temporary_file(root_dir=pex_chroot) as fp:
+      fp.write(b'import sys; print(" ".join(sys.argv))')
+      fp.close()
+      process = PEX(pex_chroot).run(args=[fp.name, 'one', 'two'],
+                                    stdout=subprocess.PIPE,
+                                    stderr=subprocess.PIPE,
+                                    blocking=False)
+      stdout, stderr = process.communicate()
+
+      assert 0 == process.returncode
+      assert '{} one two\n'.format(fp.name).encode('utf-8') == stdout
+      assert b'' == stderr
+
+
 def test_pex_run_custom_setuptools_useable():
   with temporary_dir() as resolve_cache:
     dists = resolve(['setuptools==36.2.7'], cache=resolve_cache)