Repository: pyodide/pyodide

Issue:
Standalone, dependency-free script to create pure python pyodide package
As a downstream user, I have some pure python packages that I want to ship with pyodide, and I would want this process to be as straightforward as possible.

Currently, to create the package suitable for pyodide consumption, I need to have quite a bit of the pyodide source code available. In particular, it requires various emscripten tools to be present (see also https://github.com/iodide-project/pyodide/issues/228#issuecomment-434940415).

In reality, the amount of code needed to produce this python package is fairly minimal --- one concatenates all the files in the python package, and then write some metadata and glue into the corresponding javascript file. This is especially true if we forgo lz4 compression (experimentally, lz4 results in more bytes transferred across the wire anyway).

I have written a short python script to do exactly that: https://github.com/SpectralSequences/sseq/blob/master/chart/python/packager.py . This is tuned fairly specificly to my package, but it is not difficult to generalize slightly to make it suitable for generic pure python packages. It would be great if a similar script is available from upstream.

I think the most reasonable way forward is to tidy up `file_packager.py` to avoid external dependencies, and instruct users on how to use it. The main changes required would be

1. Vendor the `shared.JS.escape_for_js_string` function that is currently imported from emscripten's scripts
2. Load the `tools.jsrun.run_js` function only when lz4 compression is enabled (this is used to run the lz4-compression script, which is written in javascript)

If there is no intention for our `file_packager.py` to track upstream, various config options can be eliminated to simplify maintenance. 

I would be happy to implement this.




Discussion:

Resolving patch:
diff --git a/tools/file_packager.py b/tools/file_packager.py
old mode 100644
new mode 100755
--- a/tools/file_packager.py
+++ b/tools/file_packager.py
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+
 # flake8: noqa
 
 # This is forked from emscripten 1.38.22, with the original copyright notice
@@ -72,20 +74,7 @@
 import uuid
 import ctypes
 
-emscripten_dir = os.path.join(
-  os.path.dirname(os.path.dirname(os.path.abspath(__file__))),
-  'emsdk', 'emsdk', 'fastcomp', 'emscripten'
-)
-tag_dir = sorted(os.listdir(emscripten_dir), key=lambda x: len(x))[0]
-sys.path.insert(1, emscripten_dir)
-
-from tools.toolchain_profiler import ToolchainProfiler
-if __name__ == '__main__':
-  ToolchainProfiler.record_process_start()
-
 import posixpath
-from tools import shared
-from tools.jsrun import run_js
 from subprocess import PIPE
 import fnmatch
 import json
@@ -251,6 +240,10 @@
     }
 '''
 
+def escape_for_js_string(s):
+  s = s.replace('\\', '/').replace("'", "\\'").replace('"', '\\"')
+  return s
+
 
 def has_hidden_attribute(filepath):
   """Win32 code to test whether the given file has the hidden property set."""
@@ -534,9 +527,18 @@ def was_seen(name):
           }
     '''
     use_data += ("          Module['removeRunDependency']('datafile_%s');\n"
-                 % shared.JS.escape_for_js_string(data_target))
+                 % escape_for_js_string(data_target))
 
   else:
+    emscripten_dir = os.path.join(
+      os.path.dirname(os.path.dirname(os.path.abspath(__file__))),
+      'emsdk', 'emsdk', 'fastcomp', 'emscripten'
+    )
+    sys.path.insert(1, emscripten_dir)
+
+    from tools import shared
+    from tools.jsrun import run_js
+
     # LZ4FS usage
     temp = data_target + '.orig'
     shutil.move(data_target, temp)
@@ -551,7 +553,7 @@ def was_seen(name):
           assert(typeof Module.LZ4 === 'object', 'LZ4 not present - was your app build with  -s LZ4=1  ?');
           Module.LZ4.loadPackage({ 'metadata': metadata, 'compressedData': compressedData });
           Module['removeRunDependency']('datafile_%s');
-    ''' % (meta, shared.JS.escape_for_js_string(data_target))
+    ''' % (meta, escape_for_js_string(data_target))
 
   package_uuid = uuid.uuid4()
   package_name = data_target
@@ -574,8 +576,8 @@ def was_seen(name):
       err('warning: you defined Module.locateFilePackage, that has been renamed to Module.locateFile (using your locateFilePackage for now)');
     }
     var REMOTE_PACKAGE_NAME = Module['locateFile'] ? Module['locateFile'](REMOTE_PACKAGE_BASE, '') : REMOTE_PACKAGE_BASE;
-  ''' % (shared.JS.escape_for_js_string(data_target),
-         shared.JS.escape_for_js_string(remote_package_name))
+  ''' % (escape_for_js_string(data_target),
+         escape_for_js_string(remote_package_name))
   metadata['remote_package_size'] = remote_package_size
   metadata['package_uuid'] = str(package_uuid)
   ret += '''
@@ -802,7 +804,7 @@ def was_seen(name):
       %s
     };
     Module['addRunDependency']('datafile_%s');
-  ''' % (use_data, shared.JS.escape_for_js_string(data_target))
+  ''' % (use_data, escape_for_js_string(data_target))
   # use basename because from the browser's point of view,
   # we need to find the datafile in the same dir as the html file
 


Test patch: