Repository: ros-infrastructure/rosdep

Issue:
Can't choose whether or not to install pip2 or pip3 version of pip package
Have run into the case where I need a specific version of a pip package, but it looks like the in the rosdep yaml files rosdep is not setup to look explicitly for either `pip2` or `pip3` and instead just looks for `pip` and uses whatever version the `pip` executable is using. It would be awesome if we could explicitly say a key should be installed with `pip2 ` or `pip3`

Discussion:
sloretz: Here's a manual test plan. I'll edit this comment with the results after I get the travis jobs to pass.

Create `/tmp/dummypkg/package.xml` with the following content

<details><summary>dummypkg/package.xml</summary>

```xml
<?xml version="1.0"?>
<package format="3">
  <name>dummypkg</name>
  <version>0.1.0</version>
  <description>empty</description>

  <license>BSD</license>

  <maintainer email="sloretz@osrfoundation.org">Shane Loretz</maintainer>

  <test_depend>python-flatdict-pip</test_depend>
</package>
```
</details>

Setup: Install rosdep from source
```
sudo python2 setup.py develop
sudo python3 setup.py develop
```

**Test 1: Check using pip3 from python3**

```
ROS_PYTHON_VERSION=3 python3 scripts/rosdep check --from-paths /tmp/dummypkg/
ROS_PYTHON_VERSION=3 python3 scripts/rosdep install --from-paths /tmp/dummypkg/
sudo pip3 uninstall -y flatdict
```
<details><summary>Test 1 results TODO</summary>

```xml

```
</details>

**Test 2: Check using pip3 from python2**
```
ROS_PYTHON_VERSION=3 python2 scripts/rosdep check --from-paths /tmp/dummypkg/
ROS_PYTHON_VERSION=3 python2 scripts/rosdep install --from-paths /tmp/dummypkg/
sudo pip3 uninstall -y flatdict
```
<details><summary>Test 2 results TODO</summary>

```xml

```
</details>

**Test 3: Check using pip2 from python2**

```
ROS_PYTHON_VERSION=2 python2 scripts/rosdep check --from-paths /tmp/dummypkg/
ROS_PYTHON_VERSION=2 python2 scripts/rosdep install --from-paths /tmp/dummypkg/
sudo pip2 uninstall -y flatdict
```
<details><summary>Test 3 results TODO</summary>

```xml
```
</details>

**Test 4: Check using pip2 from python3**
```
ROS_PYTHON_VERSION=2 python3 scripts/rosdep check --from-paths /tmp/dummypkg/
ROS_PYTHON_VERSION=2 python3 scripts/rosdep install --from-paths /tmp/dummypkg/
sudo pip2 uninstall -y flatdict
```

<details><summary>Test 4 results TODO</summary>

```xml

```
</details>

**Setup: Get rid of pip2 and pip3 executables for the rest of the tests**
```
sudo mv $(which pip2) $(which pip2).back
sudo mv $(which pip3) $(which pip3).back
```

**Test 5: Check using sys.executable from python3**
```
ROS_PYTHON_VERSION=3 python3 scripts/rosdep check --from-paths /tmp/dummypkg/
ROS_PYTHON_VERSION=3 python3 scripts/rosdep install --from-paths /tmp/dummypkg/
sudo pip3.back uninstall -y flatdict
```

<details><summary>Test 5 results TODO</summary>

```xml

```
</details>

**Test 6: Check using sys.executable from python2**

```
ROS_PYTHON_VERSION=2 python2 scripts/rosdep check --from-paths /tmp/dummypkg/
ROS_PYTHON_VERSION=2 python2 scripts/rosdep install --from-paths /tmp/dummypkg/
sudo pip2.back uninstall -y flatdict
```
<details><summary>Test 6 results TODO</summary>

```xml

```
</details>

**Test 7: Check using python2 from python3**

```
ROS_PYTHON_VERSION=2 python3 scripts/rosdep check --from-paths /tmp/dummypkg/
ROS_PYTHON_VERSION=2 python3 scripts/rosdep install --from-paths /tmp/dummypkg/
sudo pip2.back uninstall -y flatdict
```

<details><summary>Test 7 results TODO</summary>

```xml

```
</details>

**Test 8: Check using python3 from python2**

```
ROS_PYTHON_VERSION=3 python2 scripts/rosdep check --from-paths /tmp/dummypkg/
ROS_PYTHON_VERSION=3 python2 scripts/rosdep install --from-paths /tmp/dummypkg/
sudo pip3.back uninstall -y flatdict
```
<details><summary>Test 8 results TODO</summary>

```xml

```
</details>
codecov-io: # [Codecov](https://codecov.io/gh/ros-infrastructure/rosdep/pull/709?src=pr&el=h1) Report
> Merging [#709](https://codecov.io/gh/ros-infrastructure/rosdep/pull/709?src=pr&el=desc) into [master](https://codecov.io/gh/ros-infrastructure/rosdep/commit/39cf7252405afb27bcfd11dc98e5edc6f801e2ef?src=pr&el=desc) will **decrease** coverage by `0.08%`.
> The diff coverage is `59.25%`.

[![Impacted file tree graph](https://codecov.io/gh/ros-infrastructure/rosdep/pull/709/graphs/tree.svg?width=650&token=YdnUPit6vO&height=150&src=pr)](https://codecov.io/gh/ros-infrastructure/rosdep/pull/709?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master     #709      +/-   ##
==========================================
- Coverage   75.82%   75.73%   -0.09%     
==========================================
  Files          38       38              
  Lines        3032     3054      +22     
==========================================
+ Hits         2299     2313      +14     
- Misses        733      741       +8
```


| [Impacted Files](https://codecov.io/gh/ros-infrastructure/rosdep/pull/709?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/rosdep2/main.py](https://codecov.io/gh/ros-infrastructure/rosdep/pull/709/diff?src=pr&el=tree#diff-c3JjL3Jvc2RlcDIvbWFpbi5weQ==) | `49.53% <100%> (+0.18%)` | :arrow_up: |
| [src/rosdep2/platforms/pip.py](https://codecov.io/gh/ros-infrastructure/rosdep/pull/709/diff?src=pr&el=tree#diff-c3JjL3Jvc2RlcDIvcGxhdGZvcm1zL3BpcC5weQ==) | `63.51% <56%> (-1.31%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/ros-infrastructure/rosdep/pull/709?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/ros-infrastructure/rosdep/pull/709?src=pr&el=footer). Last update [39cf725...0ae5817](https://codecov.io/gh/ros-infrastructure/rosdep/pull/709?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Patch:
diff --git a/CHANGELOG.rst b/CHANGELOG.rst
index 74b3bc3..acd4d40 100644
--- a/CHANGELOG.rst
+++ b/CHANGELOG.rst
@@ -1,3 +1,10 @@
+0.16.2 (2019-10-18)
+-------------------
+- Bump rospkg dependency to guarantee all platforms are available..
+  - https://github.com/ros-infrastructure/rosdep/issues/717
+- Set ROS_PYTHON_VERSION if unset.
+  - https://github.com/ros-infrastructure/rosdep/issues/708
+
 0.16.1 (2019-09-19)
 -------------------
 
diff --git a/setup.py b/setup.py
index 59ec4ea..cfe9ae6 100644
--- a/setup.py
+++ b/setup.py
@@ -8,7 +8,7 @@ setup(
     version=__version__,  # noqa:F821
     packages=['rosdep2', 'rosdep2.ament_packages', 'rosdep2.platforms'],
     package_dir={'': 'src'},
-    install_requires=['catkin_pkg >= 0.4.0', 'rospkg >= 1.1.8', 'rosdistro >= 0.7.5', 'PyYAML >= 3.1'],
+    install_requires=['catkin_pkg >= 0.4.0', 'rospkg >= 1.1.10', 'rosdistro >= 0.7.5', 'PyYAML >= 3.1'],
     test_suite='nose.collector',
     test_requires=['mock', 'nose >= 1.0'],
     scripts=['scripts/rosdep', 'scripts/rosdep-source'],
diff --git a/src/rosdep2/_version.py b/src/rosdep2/_version.py
index 9513287..e935064 100644
--- a/src/rosdep2/_version.py
+++ b/src/rosdep2/_version.py
@@ -1,1 +1,1 @@
-__version__ = '0.16.1'
+__version__ = '0.16.2'
diff --git a/src/rosdep2/platforms/pip.py b/src/rosdep2/platforms/pip.py
index 7ae8ffd..65ab596 100644
--- a/src/rosdep2/platforms/pip.py
+++ b/src/rosdep2/platforms/pip.py
@@ -29,8 +29,10 @@
 
 from __future__ import print_function
 
+import os
 import pkg_resources
 import subprocess
+import sys
 
 from ..core import InstallFailed
 from ..installers import PackageManagerInstaller
@@ -44,9 +46,31 @@ def register_installers(context):
     context.set_installer(PIP_INSTALLER, PipInstaller())
 
 
-def is_pip_installed():
+def get_pip_command():
+    # First try pip2 or pip3
+    cmd = ['pip' + os.environ['ROS_PYTHON_VERSION']]
+    if is_cmd_available(cmd):
+        return cmd
+
+    # Second, try using the same python executable since we know that exists
+    if os.environ['ROS_PYTHON_VERSION'] == sys.version[0]:
+        try:
+            import pip
+        except ImportError:
+            pass
+        else:
+            return [sys.executable, '-m', 'pip']
+
+    # Finally, try python2 or python3 commands
+    cmd = ['python' + os.environ['ROS_PYTHON_VERSION'], '-m', 'pip']
+    if is_cmd_available(cmd):
+        return cmd
+    return None
+
+
+def is_cmd_available(cmd):
     try:
-        subprocess.Popen(['pip'], stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()
+        subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()
         return True
     except OSError:
         return False
@@ -58,11 +82,15 @@ def pip_detect(pkgs, exec_fn=None):
 
     :param exec_fn: function to execute Popen and read stdout (for testing)
     """
+    pip_cmd = get_pip_command()
+    if not pip_cmd:
+        return []
+
     fallback_to_pip_show = False
     if exec_fn is None:
         exec_fn = read_stdout
         fallback_to_pip_show = True
-    pkg_list = exec_fn(['pip', 'freeze']).split('\n')
+    pkg_list = exec_fn(pip_cmd + ['freeze']).split('\n')
 
     ret_list = []
     for pkg in pkg_list:
@@ -79,7 +107,7 @@ def pip_detect(pkgs, exec_fn=None):
         for pkg in [p for p in pkgs if p not in ret_list]:
             # does not see retcode but stdout for old pip to check if installed
             proc = subprocess.Popen(
-                ['pip', 'show', pkg],
+                pip_cmd + ['show', pkg],
                 stdout=subprocess.PIPE,
                 stderr=subprocess.STDOUT
             )
@@ -110,12 +138,13 @@ class PipInstaller(PackageManagerInstaller):
         return version_strings
 
     def get_install_command(self, resolved, interactive=True, reinstall=False, quiet=False):
-        if not is_pip_installed():
+        pip_cmd = get_pip_command()
+        if not pip_cmd:
             raise InstallFailed((PIP_INSTALLER, 'pip is not installed'))
         packages = self.get_packages_to_install(resolved, reinstall=reinstall)
         if not packages:
             return []
-        cmd = ['pip', 'install', '-U']
+        cmd = pip_cmd + ['install', '-U']
         if quiet:
             cmd.append('-q')
         if reinstall:
diff --git a/stdeb.cfg b/stdeb.cfg
index 725c84a..7185cf7 100644
--- a/stdeb.cfg
+++ b/stdeb.cfg
@@ -1,6 +1,6 @@
 [DEFAULT]
-Depends: ca-certificates, python-rospkg (>= 1.1.8), python-yaml, python-catkin-pkg (>= 0.4.0), python-rosdistro (>= 0.7.5), sudo
-Depends3: ca-certificates, python3-rospkg (>= 1.1.8), python3-yaml, python3-catkin-pkg (>= 0.4.0), python3-rosdistro (>= 0.7.5), sudo
+Depends: ca-certificates, python-rospkg (>= 1.1.10), python-yaml, python-catkin-pkg (>= 0.4.0), python-rosdistro (>= 0.7.5), sudo
+Depends3: ca-certificates, python3-rospkg (>= 1.1.10), python3-yaml, python3-catkin-pkg (>= 0.4.0), python3-rosdistro (>= 0.7.5), sudo
 Conflicts: python3-rosdep, python-rosdep2, python3-rosdep2
 Conflicts3: python-rosdep, python-rosdep2, python3-rosdep2
 Copyright-File: LICENSE


Test patch:
diff --git a/test/test_rosdep_pip.py b/test/test_rosdep_pip.py
index 9013d54..ad86aae 100644
--- a/test/test_rosdep_pip.py
+++ b/test/test_rosdep_pip.py
@@ -28,6 +28,7 @@
 # Author Ken Conley/kwc@willowgarage.com
 
 import os
+import sys
 import traceback
 from mock import Mock, patch
 
@@ -70,9 +71,9 @@ def test_PipInstaller():
     from rosdep2 import InstallFailed
     from rosdep2.platforms.pip import PipInstaller
 
-    @patch('rosdep2.platforms.pip.is_pip_installed')
+    @patch('rosdep2.platforms.pip.get_pip_command')
     def test_no_pip(mock_method):
-        mock_method.return_value = False
+        mock_method.return_value = None
         try:
             installer = PipInstaller()
             installer.get_install_command(['whatever'])
@@ -82,22 +83,22 @@ def test_PipInstaller():
 
     test_no_pip()
 
-    @patch('rosdep2.platforms.pip.is_pip_installed')
+    @patch('rosdep2.platforms.pip.get_pip_command')
     @patch.object(PipInstaller, 'get_packages_to_install')
-    def test(mock_method, mock_is_pip_installed):
-        mock_is_pip_installed.return_value = True
+    def test(mock_method, mock_get_pip_command):
+        mock_get_pip_command.return_value = ['mock-pip']
         installer = PipInstaller()
         mock_method.return_value = []
         assert [] == installer.get_install_command(['fake'])
 
         # no interactive option with PIP
         mock_method.return_value = ['a', 'b']
-        expected = [['sudo', '-H', 'pip', 'install', '-U', 'a'],
-                    ['sudo', '-H', 'pip', 'install', '-U', 'b']]
+        expected = [['sudo', '-H', 'mock-pip', 'install', '-U', 'a'],
+                    ['sudo', '-H', 'mock-pip', 'install', '-U', 'b']]
         val = installer.get_install_command(['whatever'], interactive=False)
         assert val == expected, val
-        expected = [['sudo', '-H', 'pip', 'install', '-U', 'a'],
-                    ['sudo', '-H', 'pip', 'install', '-U', 'b']]
+        expected = [['sudo', '-H', 'mock-pip', 'install', '-U', 'a'],
+                    ['sudo', '-H', 'mock-pip', 'install', '-U', 'b']]
         val = installer.get_install_command(['whatever'], interactive=True)
         assert val == expected, val
     try:
@@ -105,3 +106,41 @@ def test_PipInstaller():
     except AssertionError:
         traceback.print_exc()
         raise
+
+
+def test_get_pip_command():
+    from rosdep2.platforms.pip import get_pip_command
+
+    # pip2 or pip3
+    @patch('rosdep2.platforms.pip.is_cmd_available')
+    def test_pip2_or_pip3(mock_is_cmd_available):
+        mock_is_cmd_available.return_value = True
+
+        with patch.dict(os.environ, {'ROS_PYTHON_VERSION': '2'}):
+            assert ['pip2'] == get_pip_command()
+
+        with patch.dict(os.environ, {'ROS_PYTHON_VERSION': '3'}):
+            assert ['pip3'] == get_pip_command()
+
+    # sys.executable (assumes pip is installed)
+    @patch('rosdep2.platforms.pip.is_cmd_available')
+    def test_sys_executable(mock_is_cmd_available):
+        mock_is_cmd_available.return_value = False
+
+        with patch.dict(os.environ, {'ROS_PYTHON_VERSION': str(sys.version[0])}):
+            assert [sys.executable, '-m', 'pip'] == get_pip_command()
+
+    # python2 or python3
+    def fake_is_cmd_available(cmd):
+        if cmd[0] in ['pip2', 'pip3']:
+            return False
+        return True
+
+    @patch('rosdep2.platforms.pip.is_cmd_available', new=fake_is_cmd_available)
+    def test_python2_or_python3(mock_is_cmd_available):
+
+        with patch.dict(os.environ, {'ROS_PYTHON_VERSION': '2'}):
+            assert ['python2', '-m', 'pip'] == get_pip_command()
+
+        with patch.dict(os.environ, {'ROS_PYTHON_VERSION': '3'}):
+            assert ['python3', '-m', 'pip'] == get_pip_command()