Repository: desihub/desiutil

Issue:
desiInstall of specex/main and fiberassign/main, which need compilation too
desiInstall supports "in-place" installations of python repos that adds `$PRODUCT_DIR/py` to `$PYTHONPATH` and `$PRODUCT_DIR/bin` to `$PATH` so that any changes to the repo are automatically available without having to do a separate installation step.  Good.

specex and fiberassign, however, are hybrid repos that have python code with compiled extensions.  An in-place install is handy when making changes to any of the python code, but if any of the C++ code changes it still has to be compiled using:
```
python setup.py build_ext --inplace
```

desiInstall doesn't know this, and this pattern doesn't fit any of the build types listed at https://desiutil.readthedocs.io/en/latest/desiInstall.html#determine-build-type .

What's the best way to get desiInstall to know that it needs to run this extra step for these two repos?

A somewhat hacky solution that may not require changing desiInstall is to leverage its special case of looking for an `etc/{productname}_data.sh` script and executing that, e.g. as used by desimodel to get the data from svn.  specex and fiberassign could add their own `etc/*_data.sh` scripts to run `python setup.py build_ext --inplace`, but that is somewhat cryptically using a data-download hook for other purposes.

It might be better to define another hook similar to `etc/*_data.sh`, and if desiInstall detects that it will run it for in-place branch installations, but not for regular installations.  That requires an update to both desiInstall and the specex+fiberassign repos, but it might be more obvious and maintainable in the future.

For context, both specex and fiberassign used to have a Makefile that desiInstall knew to run, but both have migrated to a python-first approach with compiled extensions without a Makefile.  Current master (now main) installations have bootstrapped the `python setup.py build_ext --inplace` upon first installation, after which the desitest nightly update cronjob re-runs that every night after `git pull`.  The point of this ticket is so that the end-user doesn't have to remember to do special steps whenever they make a fresh main installation.

@weaverba137 thoughts?

Discussion:

Patch:
diff --git a/doc/changes.rst b/doc/changes.rst
index 8697970..93581d9 100644
--- a/doc/changes.rst
+++ b/doc/changes.rst
@@ -8,9 +8,14 @@ Change Log
 * :command:`desiInstall` uses desihub location of simqso fork (commit e963344_).
 * Allow :command:`desiInstall` to remove permission-locked directories;
   suppress certain :command:`pip` warnings (PR `#185`_).
+* Allow :command:`desiInstall` to compile code in certain branch installs (PR `#188`_).
+* Add `gpu_specter`_ to known packages (PR `#189`_).
 
 .. _e963344: https://github.com/desihub/desiutil/commit/e963344cd072255174187d2bd6da72d085745abd
 .. _`#185`: https://github.com/desihub/desiutil/pull/185
+.. _`#188`: https://github.com/desihub/desiutil/pull/188
+.. _`#189`: https://github.com/desihub/desiutil/pull/189
+.. _`gpu_specter`: https://github.com/desihub/gpu_specter
 
 3.2.5 (2022-01-20)
 ------------------
diff --git a/doc/desiInstall.rst b/doc/desiInstall.rst
index e433192..8c926f0 100644
--- a/doc/desiInstall.rst
+++ b/doc/desiInstall.rst
@@ -322,6 +322,23 @@ not bundled with the code.  The script should download data *directly* to
 with :command:`desiInstall` and unit tests.  Note that here are other, better ways to
 install and manipulate data that is bundled *with* a Python package.
 
+Compile in Branch Installs
+--------------------------
+
+In a few cases (fiberassign_, specex_) code needs to be compiled even when
+installing a branch.  If :command:`desiInstall` detects a branch install *and*
+the script ``etc/product_compile.sh`` exists, :command:`desiInstall` will run this
+script, supplying the Python executable path as a single command-line argument.
+The script itself is intended to be a thin wrapper on *e.g.*::
+
+    #!/bin/bash
+    py=$1
+    ${py} setup.py build_ext --inplace
+
+
+.. _fiberassign: https://github.com/desihub/fiberassign
+.. _specex: https://github.com/desihub/specex
+
 Fix Permissions
 ---------------
 
diff --git a/py/desiutil/install.py b/py/desiutil/install.py
index 28c92ad..7dfc4cf 100644
--- a/py/desiutil/install.py
+++ b/py/desiutil/install.py
@@ -910,6 +910,33 @@ class DesiInstall(object):
                     raise DesiInstallException(message)
         return
 
+    def compile_branch(self):
+        """Certain packages need C/C++ code compiled even for a branch install.
+        """
+        if self.is_branch:
+            compile_script = os.path.join(self.install_dir, 'etc',
+                                          '{0}_compile.sh'.format(self.baseproduct))
+            if os.path.exists(compile_script):
+                self.log.debug("Detected compile script: %s.", compile_script)
+                if self.options.test:
+                    self.log.debug('Test Mode. Skipping compile script.')
+                else:
+                    current_dir = os.getcwd()
+                    self.log.debug("os.chdir('%s')", self.install_dir)
+                    os.chdir(self.install_dir)
+                    proc = Popen([compile_script, sys.executable], universal_newlines=True,
+                                 stdout=PIPE, stderr=PIPE)
+                    out, err = proc.communicate()
+                    status = proc.returncode
+                    self.log.debug(out)
+                    self.log.debug("os.chdir('%s')", current_dir)
+                    os.chdir(current_dir)
+                    if status != 0 and len(err) > 0:
+                        message = "Error compiling code: {0}".format(err)
+                        self.log.critical(message)
+                        raise DesiInstallException(message)
+        return
+
     def verify_bootstrap(self):
         """Make sure that desiutil/desiInstall was installed with
         an explicit Python executable path.
@@ -1027,6 +1054,7 @@ class DesiInstall(object):
             self.prepare_environment()
             self.install()
             self.get_extra()
+            self.compile_branch()
             self.verify_bootstrap()
             self.permissions()
         except DesiInstallException:


Test patch:
diff --git a/py/desiutil/test/test_install.py b/py/desiutil/test/test_install.py
index 73c18f7..a12433f 100644
--- a/py/desiutil/test/test_install.py
+++ b/py/desiutil/test/test_install.py
@@ -2,6 +2,7 @@
 # -*- coding: utf-8 -*-
 """Test desiutil.install.
 """
+import sys
 import unittest
 from unittest.mock import patch, call, MagicMock, mock_open
 from os import chdir, environ, getcwd, mkdir, remove, rmdir
@@ -507,6 +508,42 @@ class TestInstall(unittest.TestCase):
         self.assertLog(-1, message)
         self.assertEqual(str(cm.exception), message)
 
+    @patch('os.chdir')
+    @patch('os.path.exists')
+    @patch('desiutil.install.Popen')
+    def test_compile_branch(self, mock_popen, mock_exists, mock_chdir):
+        """Test compiling code in certain cases.
+        """
+        current_dir = getcwd()
+        options = self.desiInstall.get_options(['fiberassign', 'branches/main'])
+        self.desiInstall.baseproduct = 'fiberassign'
+        self.desiInstall.is_branch = True
+        self.desiInstall.install_dir = join(self.data_dir, 'fiberassign')
+        mock_exists.return_value = True
+        mock_proc = mock_popen()
+        mock_proc.returncode = 0
+        mock_proc.communicate.return_value = ('out', 'err')
+        self.desiInstall.compile_branch()
+        mock_chdir.assert_has_calls([call(self.desiInstall.install_dir),
+                                     call(current_dir)])
+        mock_exists.assert_has_calls([call(join(self.desiInstall.install_dir, 'etc', 'fiberassign_compile.sh'))])
+        mock_popen.assert_has_calls([call([join(self.desiInstall.install_dir, 'etc', 'fiberassign_compile.sh'), sys.executable],
+                                          stderr=-1, stdout=-1, universal_newlines=True)], any_order=True)
+        mock_popen.reset_mock()
+        self.desiInstall.options.test = True
+        self.desiInstall.compile_branch()
+        self.assertLog(-1, 'Test Mode. Skipping compile script.')
+        mock_popen.reset_mock()
+        self.desiInstall.options.test = False
+        mock_proc = mock_popen()
+        mock_proc.returncode = 1
+        mock_proc.communicate.return_value = ('out', 'err')
+        with self.assertRaises(DesiInstallException) as cm:
+            self.desiInstall.compile_branch()
+        message = "Error compiling code: err"
+        self.assertLog(-1, message)
+        self.assertEqual(str(cm.exception), message)
+
     def test_verify_bootstrap(self):
         """Test proper installation of the desiInstall executable.
         """