Repository: modin-project/modin
Pull request: FEAT-#2138: sync modin version between local and remote contexts
Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

<!--
Thank you for your contribution! 
Please review the contributing docs: https://modin.readthedocs.io/en/latest/contributing.html
if you have questions about contributing.
-->

## What do these changes do?

<!-- Please give a short brief about these changes. -->

- [x] commit message follows format outlined [here](https://modin.readthedocs.io/en/latest/contributing.html)
- [x] passes `flake8 modin`
- [x] passes `black --check modin`
- [x] signed commit with `git commit -s` <!-- you can amend your commit with a signature via `git commit -amend -s` -->
- [x] Resolves #2138 <!-- issue must be created for each patch -->
- [ ] tests added and passing


Problem statement:
Sync modin version between local and remote contexts
**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. What kind of performance improvements would you like to see with this new API?

Hints from issue discussion:

Commit message: FEAT-#2138: fix

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

Patch to review:
diff --git a/modin/experimental/cloud/ray-autoscaler.yml b/modin/experimental/cloud/ray-autoscaler.yml
index 0d022578ae1..52d9079d49c 100644
--- a/modin/experimental/cloud/ray-autoscaler.yml
+++ b/modin/experimental/cloud/ray-autoscaler.yml
@@ -120,7 +120,8 @@ setup_commands:
         source ~/.bashrc
         conda create --name modin --yes
         conda activate modin
-        conda install --yes --override-channels -c intel/label/validation -c conda-forge modin {{CONDA_PACKAGES}}
+        conda install --yes --override-channels -c intel/label/validation -c conda-forge {{CONDA_PACKAGES}}
+        {{INSTALL_MODIN_COMMAND}}
 
         pip install colorful cloudpickle
         # ray now executes "ray stop" which expects "ray" to be in $PATH
@@ -138,7 +139,6 @@ head_setup_commands:
     - |
         source ~/.bashrc
         conda activate modin
-        pip install boto3==1.4.8 rpyc==4.1.5
         echo 'export MODIN_REDIS_ADDRESS="localhost:6379"' >> ~/.bashrc
 
 # Custom commands that will be run on worker nodes after common setup.
diff --git a/modin/experimental/cloud/rayscale.py b/modin/experimental/cloud/rayscale.py
index e33b5dcea4b..50725078a76 100644
--- a/modin/experimental/cloud/rayscale.py
+++ b/modin/experimental/cloud/rayscale.py
@@ -13,11 +13,13 @@
 
 import threading
 import os
+import re
 import traceback
 import sys
 from hashlib import sha1
 from typing import Callable
 import subprocess
+import warnings
 
 import yaml
 from ray.autoscaler.commands import (
@@ -158,10 +160,14 @@ def _conda_requirements(self):
         return reqs_with_quotes
 
     def _update_conda_requirements(self, setup_commands: str):
-        return setup_commands.replace(
+        setup_commands = setup_commands.replace(
             "{{CONDA_PACKAGES}}", " ".join(self._conda_requirements())
         )
 
+        return setup_commands.replace(
+            "{{INSTALL_MODIN_COMMAND}}", self._get_modin_install_command()
+        )
+
     @staticmethod
     def _get_python_version():
         major = sys.version_info.major
@@ -169,6 +175,93 @@ def _get_python_version():
         micro = sys.version_info.micro
         return [f"python>={major}.{minor}", f"python<={major}.{minor}.{micro}"]
 
+    @staticmethod
+    def _git_state():
+        """
+        Suppose git in PATH.
+        """
+        import modin
+
+        cwd = os.path.dirname(modin.__file__)
+        git_branch_vv = subprocess.check_output(
+            ["git", "branch", "-vv"], cwd=cwd, encoding="utf-8"
+        ).strip("\n")
+        git_remote_v = subprocess.check_output(
+            ["git", "remote", "-v"], cwd=cwd, encoding="utf-8"
+        ).strip("\n")
+
+        local_branch = None
+        remote_branch = None
+        for line in re.split(r"\n+", git_branch_vv):
+            # * LOCAL_BRANCH HASH [REPO_ALIAS/REMOTE_BRANCH[: STATUS]] [Commit message]
+            if line.startswith("*"):
+                local_branch = re.split(r"\s+", line)[1]
+                if local_branch == "HEAD":
+                    # example case: (HEAD detached at modin/sync-modin-between-contexts)
+                    raise ValueError("You are in 'detached HEAD' state")
+                if "[" not in line:
+                    raise ValueError(
+                        f"local branch: [{local_branch}] does not track remote"
+                    )
+                dirty_remote_branch = re.split(r"\s+", line)[3]
+                # remove "(" and ":" or ")"
+                remote_branch = dirty_remote_branch[1:][:-1]
+                break
+
+        try:
+            repo_alias, remote_branch = remote_branch.split("/", maxsplit=1)
+        except ValueError:
+            raise ValueError(
+                f"The remote branch: [{remote_branch}] obtained from \
+                git: [{git_branch_vv}] for local_branch: [{local_branch}] is not valid"
+            )
+
+        repo_name = None
+        for line in re.split(r"\n+", git_remote_v):
+            # repo_alias repo_name (fetch/push)
+            if line.startswith(repo_alias):
+                repo_name = re.split(r"\s+", line)[1]
+                break
+
+        if repo_name is None:
+            raise ValueError(
+                f"git remote -v: [{git_remote_v}] doesn't include info for repo alias: [{repo_alias}]"
+            )
+
+        return repo_name, remote_branch
+
+    @staticmethod
+    def _get_modin_install_command():
+        from modin import __version__
+
+        if len(__version__.split("+")) == 1:
+            # example version: 0.8.0
+            return f"conda install --yes --override-channels -c intel/label/validation \
+                -c conda-forge modin=={__version__}"
+        else:
+            # example version: 0.8.0+103.gfe0afed.dirty
+            # currently install only from last commit in any branch
+            try:
+                repo, branch = RayCluster._git_state()
+            except Exception as er:
+                warnings.warn(str(er))
+                warnings.warn(
+                    "failed get git repo and branch; installing latest release of modin"
+                )
+                return "conda install --yes --override-channels -c intel/label/validation -c conda-forge modin"
+
+            modin_install = f"""
+        sudo apt-get update -y
+        sudo apt-get install -y build-essential
+
+        rm -Rf modin
+        git clone --single-branch --depth 1 --branch {branch} {repo}
+        (cd modin && pip install -e .[ray] --use-feature=2020-resolver)
+        (cd modin && pip install -e .[remote] --use-feature=2020-resolver)
+        (cd modin && pip install -e . --use-feature=2020-resolver)"""
+
+        return modin_install
+
     @staticmethod
     def __save_config(config):
         cfgdir = os.path.abspath(os.path.expanduser("~/.modin/cloud"))
diff --git a/modin/experimental/cloud/test/test_cloud.py b/modin/experimental/cloud/test/test_cloud.py
index f35e08f5fbd..e4394685a50 100644
--- a/modin/experimental/cloud/test/test_cloud.py
+++ b/modin/experimental/cloud/test/test_cloud.py
@@ -50,3 +50,31 @@ def test_update_conda_requirements(setup_commands_source):
     )
     assert "scikit-learn>=0.23" in setup_commands_result
     assert "{{CONDA_PACKAGES}}" not in setup_commands_result
+
+
+@pytest.mark.parametrize(
+    "git_output",
+    [
+        {
+            "remote": """
+modin   https://github.com/modin-project/modin.git (fetch)
+modin   https://github.com/modin-project/modin.git (push)
+origin  https://github.com/username/modin.git (fetch)
+origin  https://github.com/username/modin.git (push)""",
+            "branch": """
+* sync-modin-between-contexts 8503e3b [origin/sync-modin-between-contexts: ahead 1] FEAT-#2138: sync modin from custom branch
+  taxi-runner                 f9bc050 [origin/taxi-runner: gone] DOCS-#1835: add runner of taxi benchmark as example
+  teamcity-coverage           3ad1674 [origin/teamcity-coverage] FIX-#9999: remove env_windows.yml
+  test-autoscaler             7d0e006 [origin/test-autoscaler: gone] add missing ray
+  update-docstrings           985adb0 Mad function implementation (#1430)""",
+        }
+    ],
+)
+def test__git_state(git_output):
+    with mock.patch(
+        "subprocess.check_output", lambda *args, **kwargs: git_output[args[0][1]]
+    ):
+        repo, branch = RayCluster._git_state()
+
+    assert repo == "https://github.com/username/modin.git"
+    assert branch == "sync-modin-between-contexts"


Code review comments:
diff_hunk: @@ -158,17 +160,108 @@ def _conda_requirements(self):
         return reqs_with_quotes
 
     def _update_conda_requirements(self, setup_commands: str):
-        return setup_commands.replace(
+        setup_commands = setup_commands.replace(
             "{{CONDA_PACKAGES}}", " ".join(self._conda_requirements())
         )
 
+        return setup_commands.replace(
+            "{{INSTALL_MODIN_COMMAND}}", self._get_modin_install_command()
+        )
+
     @staticmethod
     def _get_python_version():
         major = sys.version_info.major
         minor = sys.version_info.minor
         micro = sys.version_info.micro
         return [f"python>={major}.{minor}", f"python<={major}.{minor}.{micro}"]
 
+    @staticmethod
+    def _git_state():
+        """
+        Suppose git in PATH.
+        """
+        import modin
+
+        cwd = os.path.dirname(modin.__file__)
+        git_branch_vv = subprocess.check_output(
+            ["git", "branch", "-vv"], cwd=cwd, encoding="utf-8"
+        ).strip("\n")
+        git_remote_v = subprocess.check_output(
+            ["git", "remote", "-v"], cwd=cwd, encoding="utf-8"
+        ).strip("\n")
+
+        local_branch = None
+        remote_branch = None
+        for line in re.split(r"\n+", git_branch_vv):
+            # * LOCAL_BRANCH HASH [REPO_ALIAS/REMOTE_BRANCH[: STATUS]] [Commit message]
+            if line.startswith("*"):
+                local_branch = re.split(r"\s+", line)[1]
+                if local_branch == "HEAD":
+                    # example case: (HEAD detached at modin/sync-modin-between-contexts)
+                    raise ValueError("You are in 'detached HEAD' state")
+                if "[" not in line:
+                    raise ValueError(
+                        f"local branch: [{local_branch}] does not track remote"
+                    )
+                dirty_remote_branch = re.split(r"\s+", line)[3]
+                # remove "(" and ":" or ")"
+                remote_branch = dirty_remote_branch[1:][:-1]
+                break
+
+        try:
+            repo_alias, remote_branch = remote_branch.split("/", maxsplit=1)
+        except ValueError:
+            raise ValueError(
+                f"The remote branch: [{remote_branch}] obtained from \
+                git: [{git_branch_vv}] for local_branch: [{local_branch}] is not valid"
+            )
+
+        repo_name = None
+        for line in re.split(r"\n+", git_remote_v):
+            # repo_alias repo_name (fetch/push)
+            if line.startswith(repo_alias):
+                repo_name = re.split(r"\s+", line)[1]
+                break
+
+        if repo_name is None:
+            raise ValueError(
+                f"git remote -v: [{git_remote_v}] doesn't include info for repo alias: [{repo_alias}]"
+            )
+
+        return repo_name, remote_branch
+
+    @staticmethod
+    def _get_modin_install_command():
+        from modin import __version__
+
+        if len(__version__.split("+")) == 1:
+            # example version: 0.8.0
+            return f"conda install --yes --override-channels -c intel/label/validation \
path: modin/experimental/cloud/rayscale.py
text: @user1:
```suggestion
            return f"conda install --yes -c intel/label/validation \
```
same about `--override-channels`

diff_hunk: @@ -158,17 +160,108 @@ def _conda_requirements(self):
         return reqs_with_quotes
 
     def _update_conda_requirements(self, setup_commands: str):
-        return setup_commands.replace(
+        setup_commands = setup_commands.replace(
             "{{CONDA_PACKAGES}}", " ".join(self._conda_requirements())
         )
 
+        return setup_commands.replace(
+            "{{INSTALL_MODIN_COMMAND}}", self._get_modin_install_command()
+        )
+
     @staticmethod
     def _get_python_version():
         major = sys.version_info.major
         minor = sys.version_info.minor
         micro = sys.version_info.micro
         return [f"python>={major}.{minor}", f"python<={major}.{minor}.{micro}"]
 
+    @staticmethod
+    def _git_state():
path: modin/experimental/cloud/rayscale.py
text: @user1:
First of all, this approach is dangerous: https://stackoverflow.com/questions/3878624/how-do-i-programmatically-determine-if-there-are-uncommitted-changes/3879077#3879077
(long story short - do not parse stuff intended for humans, always try to use machine-readable output)

Second, I'm not entirely sure we need all this functionality - it's pretty easy to edit `ray-autoscaler.yml` manually during developmentto make it install modin from source, and I highly doubt any of our users would need this funcitonality (now Occam's Razor comes into play).

Let's not overcomplicate things.

@author:
Make sense!

diff_hunk: @@ -158,17 +160,108 @@ def _conda_requirements(self):
         return reqs_with_quotes
 
     def _update_conda_requirements(self, setup_commands: str):
-        return setup_commands.replace(
+        setup_commands = setup_commands.replace(
             "{{CONDA_PACKAGES}}", " ".join(self._conda_requirements())
         )
 
+        return setup_commands.replace(
+            "{{INSTALL_MODIN_COMMAND}}", self._get_modin_install_command()
+        )
+
     @staticmethod
     def _get_python_version():
         major = sys.version_info.major
         minor = sys.version_info.minor
         micro = sys.version_info.micro
         return [f"python>={major}.{minor}", f"python<={major}.{minor}.{micro}"]
 
+    @staticmethod
+    def _git_state():
+        """
+        Suppose git in PATH.
+        """
+        import modin
+
+        cwd = os.path.dirname(modin.__file__)
+        git_branch_vv = subprocess.check_output(
+            ["git", "branch", "-vv"], cwd=cwd, encoding="utf-8"
+        ).strip("\n")
+        git_remote_v = subprocess.check_output(
+            ["git", "remote", "-v"], cwd=cwd, encoding="utf-8"
+        ).strip("\n")
+
+        local_branch = None
+        remote_branch = None
+        for line in re.split(r"\n+", git_branch_vv):
+            # * LOCAL_BRANCH HASH [REPO_ALIAS/REMOTE_BRANCH[: STATUS]] [Commit message]
+            if line.startswith("*"):
+                local_branch = re.split(r"\s+", line)[1]
+                if local_branch == "HEAD":
+                    # example case: (HEAD detached at modin/sync-modin-between-contexts)
+                    raise ValueError("You are in 'detached HEAD' state")
+                if "[" not in line:
+                    raise ValueError(
+                        f"local branch: [{local_branch}] does not track remote"
+                    )
+                dirty_remote_branch = re.split(r"\s+", line)[3]
+                # remove "(" and ":" or ")"
+                remote_branch = dirty_remote_branch[1:][:-1]
+                break
+
+        try:
+            repo_alias, remote_branch = remote_branch.split("/", maxsplit=1)
+        except ValueError:
+            raise ValueError(
+                f"The remote branch: [{remote_branch}] obtained from \
+                git: [{git_branch_vv}] for local_branch: [{local_branch}] is not valid"
+            )
+
+        repo_name = None
+        for line in re.split(r"\n+", git_remote_v):
+            # repo_alias repo_name (fetch/push)
+            if line.startswith(repo_alias):
+                repo_name = re.split(r"\s+", line)[1]
+                break
+
+        if repo_name is None:
+            raise ValueError(
+                f"git remote -v: [{git_remote_v}] doesn't include info for repo alias: [{repo_alias}]"
+            )
+
+        return repo_name, remote_branch
+
+    @staticmethod
+    def _get_modin_install_command():
+        from modin import __version__
+
+        if len(__version__.split("+")) == 1:
+            # example version: 0.8.0
+            return f"conda install --yes --override-channels -c intel/label/validation \
+                -c conda-forge modin=={__version__}"
+        else:
+            # example version: 0.8.0+103.gfe0afed.dirty
+            # currently install only from last commit in any branch
+            try:
+                repo, branch = RayCluster._git_state()
+            except Exception as er:
+                warnings.warn(str(er))
+                warnings.warn(
+                    "failed get git repo and branch; installing latest release of modin"
+                )
+                return "conda install --yes --override-channels -c intel/label/validation -c conda-forge modin"
path: modin/experimental/cloud/rayscale.py
text: @user1:
```suggestion
                return "conda install --yes -c intel/label/validation -c conda-forge modin"
```

diff_hunk: @@ -120,7 +120,8 @@ setup_commands:
         source ~/.bashrc
         conda create --name modin --yes
         conda activate modin
-        conda install --yes --override-channels -c intel/label/validation -c conda-forge modin {{CONDA_PACKAGES}}
+        conda install --yes --override-channels -c intel/label/validation -c conda-forge {{CONDA_PACKAGES}}
path: modin/experimental/cloud/ray-autoscaler.yml
text: @user1:
```suggestion
        conda install --yes -c intel/label/validation -c conda-forge {{CONDA_PACKAGES}}
```
we need this `--override-channels` dropped as new version of conda package for Modin requires `libffi >= 3.3` which is available in `main` channel only