Repository: xonsh/xonsh

Issue:
pip install . doesn't install __amalgam__ and parser tables. 
It seems that `__amalgam__.py` and parser tables are not installed when running `pip install .`. The old `python setup.py install` does the right thing. 

Pip does create the "`__amalgam__.py`" files in the temporary build directory. But it doesn't copy them afterwards. Probably it ignores all temporary files generated during the build

To reproduce: 
1. Ensure that xonsh repo is clean from any "`__amalgam__.py`" files. Otherwise it will appear to work. 
2. Ensure that you remove the "Lib/site-packages/xonsh" folder before trying. 




Discussion:
Yeah this is a huge issue
Uggh pip just seems totally broken. Manually adding these to data files and the manifest did nothing
> Uggh pip just seems totally broken. Manually adding these to data files and the manifest did nothing

Are you sure, that's a problem with pip? When I did `python setup.py sdist` in a clean working directory a few days ago, the files were not in the archive either.

Edit: So it rather seems to be a problem with setuptools.
Yeah, sdists are just a tarball, so that is expected. They should appear in a wheel though. I guess I trying to figure out how to get genereated files into the wheel.
Or maybe generate the files before the wheel is created somehow?
> Yeah, sdists are just a tarball, so that is expected.

No, it's not. It should be handled by `class xsdist(sdist)`, but it's not.
Maybe it is that `get_file_list` needs to be overridden?
> Or maybe generate the files before the wheel is created somehow?

Yes. I think that will work. But I am not sure we can run code before pip copies everything to the temp build dir. Files are only copied/installed in the end if they are present in the source directory before the build starts.

 What about some dummy/empty files in the source repository. They could then be regenerated during the build as we do now. I think that will ensure they are copied. 
Actually I think we might need to change from install to build_py

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -17,6 +17,7 @@
 from setuptools.command.sdist import sdist
 from setuptools.command.install import install
 from setuptools.command.develop import develop
+from setuptools.command.build_py import build_py
 from setuptools.command.install_scripts import install_scripts
 
 try:
@@ -195,9 +196,23 @@ def restore_version():
         f.write(upd)
 
 
+class xbuild_py(build_py):
+    """Xonsh specialization of setuptools build_py class."""
+
+    def run(self):
+        clean_tables()
+        build_tables()
+        amalgamate_source()
+        # add dirty version number
+        dirty = dirty_version()
+        super().run()
+        if dirty:
+            restore_version()
+
+
 class xinstall(install):
     """Xonsh specialization of setuptools install class.
-    For production, let setuptools generate the 
+    For production, let setuptools generate the
     startup script, e.g: `pip installl .' rather than
     relying on 'python setup.py install'."""
 
@@ -217,6 +232,7 @@ def run(self):
 
             traceback.print_exc()
             print("Installing Jupyter hook failed.")
+
         super().run()
         if dirty:
             restore_version()
@@ -230,6 +246,7 @@ def make_release_tree(self, basedir, files):
         build_tables()
         amalgamate_source()
         dirty = dirty_version()
+        files.extend(TABLES)
         super().make_release_tree(basedir, files)
         if dirty:
             restore_version()
@@ -279,18 +296,15 @@ def run(self):
 
 
 # The custom install needs to be used on Windows machines
+cmdclass = {
+    "install": xinstall,
+    "sdist": xsdist,
+    "build_py": xbuild_py,
+}
 if os.name == "nt":
-    cmdclass = {
-        "install": xinstall,
-        "sdist": xsdist,
-        "install_scripts": install_scripts_quoted_shebang,
-    }
+    cmdclass["install_scripts"] = install_scripts_quoted_shebang
 else:
-    cmdclass = {
-        "install": xinstall,
-        "sdist": xsdist,
-        "install_scripts": install_scripts_rewrite,
-    }
+    cmdclass["install_scripts"] = install_scripts_rewrite
 
 
 class xdevelop(develop):


Test patch: