Repository: catkin/catkin_tools

Issue:
Potential trollius issue
[//]: # (Thank you for reporting an issue for catkin_tools!)

[//]: # (Please fill out the details below so that your issue can be resolved quickly!)
[//]: # (If you want to suggest a feature or start a discussion, feel free to delete them.)

### System Info

* Operating System: Docker container running on: `Linux 2b5aca1a9b8b 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14 08:53:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux`
* Python Version: `python --version Python 2.7.12
python3 --version Python 3.5.2`
* Version of catkin_tools: 
```
catkin --version
catkin_tools 0.4.4 (C) 2014-2018 Open Source Robotics Foundation
catkin_tools is released under the Apache License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0)
---
Using Python 3.5.2 (default, Nov 23 2017, 16:37:01) [GCC 5.4.0 20160609]
```
* ROS Distro: `kinetic`

### Build / Run Issue
* [x] I did  not `read this`

### Expected Behavior
Output build status correctly to the terminal and cleanly exit

### Actual Behavior

```bash
root@dadc0fe3d78b:/home/ws# catkin build
-------------------------------------------------------
Profile:                     default
Extending:        [explicit] /opt/ros/kinetic
Workspace:                   /home/ws
-------------------------------------------------------
Source Space:       [exists] /home/ws/src
Log Space:         [missing] /home/ws/logs
Build Space:        [exists] /home/ws/build
Devel Space:        [exists] /home/ws/devel
Install Space:     [missing] /home/ws/install
DESTDIR:            [unused] None
-------------------------------------------------------
Devel Space Layout:          linked
Install Space Layout:        merged
-------------------------------------------------------
Additional CMake Args:       -DCMAKE_BUILD_TYPE=Release
Additional Make Args:        None
Additional catkin Make Args: None
Internal Make Job Server:    True
Cache Job Environments:      False
-------------------------------------------------------
Whitelisted Packages:        None
Blacklisted Packages:        None
-------------------------------------------------------
Workspace configuration appears valid.

NOTE: Forcing CMake to run for each package.
-------------------------------------------------------
[build] Found '1' packages in 0.0 seconds.                                                                                                                                                                                                    
Starting  >>> catkin_tools_prebuild                                                                                                                                                                                                           
Finished  <<< catkin_tools_prebuild                [ 2.2 seconds ]                                                                                                                                                                            
Starting  >>> <Project_Name>                                                                                                                                                                                                                       
Finished  <<< <Project_Name>                        [ 3.3 seconds ]                                                                                                                                                                            
[build] Summary: All 2 packages succeeded!                                                                                                                                                                                                    
[build]   Ignored:   None.                                                                                                                                                                                                                    
[build]   Warnings:  None.                                                                                                                                                                                                                    
[build]   Abandoned: None.                                                                                                                                                                                                                    
[build]   Failed:    None.                                                                                                                                                                                                                    
[build] Runtime: 5.4 seconds total.                                                                                                                                                                                                           
[build] Note: Workspace packages have changed, please re-source setup files to use them.
Exception ignored in: <bound method BaseEventLoop.__del__ of <_UnixSelectorEventLoop running=False closed=True debug=False>>
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/trollius/base_events.py", line 395, in __del__
  File "/usr/local/lib/python3.5/dist-packages/trollius/unix_events.py", line 65, in close
  File "/usr/local/lib/python3.5/dist-packages/trollius/unix_events.py", line 166, in remove_signal_handler
  File "/usr/lib/python3.5/signal.py", line 47, in signal
TypeError: signal handler must be signal.SIG_IGN, signal.SIG_DFL, or a callable object
```

This error output shows up after catkin build, catkin run_tests, and catkin test. It does NOT show up after catkin_make or catkin_test_results build.

### Steps to Reproduce the Issue

We are using a modified kinetic ros docker - added pip/catkin etc. 
Single simple package, few system deps, build, run tests. 

Any idea if this is a trollius issue directly or a python2/3 issue? 

Discussion:
timonegk: A version-agnostic implementation unfortunately does not seem to be possible because the new `from()` function is invalid syntax in python2. But there there might be a way of circumventing this function that I am not aware of.
v4hn: @mikepurvis How do you want to handle this version-incompatibility here then?
Any chance of having a separate python3 branch?
mikepurvis: Thanks for doing this work! Looks like the changes are pretty straight forward.

Per @wjwwood's comment on https://github.com/catkin/catkin_tools/issues/558#issuecomment-538165609, I'd be fine to just merge this and release it as 0.5.0, supporting Python 3.5+ and Ubuntu 16.04+ only. Trusty is five years old at this point, and ROS Indigo was EOL in April 2019.

At the end of the day, `catkin_tools` is a CLI utility, not a library, so short of weirdness with ROS workspace `PYTHONPATH` values, it really shouldn't really impact people that much to have the tool suddenly be running under Python 3 rather than 2 (and if the `PYTHONPATH` does turn out to be a problem, we could consider adding logic to the entry_point that sanitizes the `sys.path` before importing anything external).
v4hn: @mikepurvis Sounds wonderful!

Could you patch CI to remove the python2.7 checks then so CI succeeds here and this gets merged?
mikepurvis: You should be able to make the CI updates part of this change! Just update the travis config here:

https://github.com/catkin/catkin_tools/blob/master/.travis.yml

And `trollius` should be dropped as a dep from the `requirements.txt` and `setup.py` files as well.
wolfv: This would be great, as this would make catkin_tools work with newer Python releases in the `conda` package manager! I just realized after I published the package that trollius and asyncio are not exactly drop-in replaceable.
wolfv: I wonder if you could use `exec(...)` or eval(...) to work around the missing keyword problem.
mikepurvis: Given that the target OS will be 16.04+, it probably makes sense to have it testing against Python 3.5, but either way this is fine for now. Thanks for the contribution!

Going to merge and then myself and others can use it from source for a bit before recommending a new release.

Patch:
diff --git a/.travis.before_install.bash b/.travis.before_install.bash
index 27eda82..a0ba725 100755
--- a/.travis.before_install.bash
+++ b/.travis.before_install.bash
@@ -6,13 +6,7 @@ if [ "$TRAVIS_OS_NAME" == "linux" ]; then
   sudo apt update
   sudo apt install enchant -y
 elif [ "$TRAVIS_OS_NAME" == "osx" ]; then
-  if [ "$PYTHON" == "/usr/local/bin/python3" ]; then
-    # Brewed Python 3.
-    brew upgrade python
-  elif [ "$PYTHON" == "/usr/local/bin/python2" ]; then
-    # Brewed Python 2.
-    brew install python@2
-  fi
+  brew upgrade python
   $PYTHON -m pip install virtualenv
   $PYTHON -m virtualenv -p $PYTHON venv
   brew install enchant
diff --git a/.travis.yml b/.travis.yml
index 7840181..c9e31cb 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -2,11 +2,6 @@
 language: generic
 matrix:
   include:
-    - dist: xenial
-      language: python
-      python: "2.7"
-      os: linux
-      env: PYTHON=/usr/bin/python2.7 CATKIN_VERSION=indigo-devel
     - dist: xenial
       language: python
       python: "3.4"
@@ -15,7 +10,7 @@ matrix:
 before_install:
   # Install catkin_tools dependencies
   - source .travis.before_install.bash
-  - pip install setuptools argparse catkin-pkg PyYAML psutil trollius osrf_pycommon pyenchant "sphinxcontrib-spelling<4.3.0"
+  - pip install setuptools argparse catkin-pkg PyYAML psutil osrf_pycommon pyenchant "sphinxcontrib-spelling<4.3.0"
 install:
   # Install catkin_tools
   - python setup.py develop
diff --git a/catkin_tools/common.py b/catkin_tools/common.py
index b04b04e..7d0cac7 100644
--- a/catkin_tools/common.py
+++ b/catkin_tools/common.py
@@ -20,7 +20,7 @@ import os
 import re
 import sys
 
-import trollius as asyncio
+import asyncio
 
 from shlex import split as _cmd_split
 try:
@@ -63,7 +63,7 @@ class FakeLock(asyncio.locks.Lock):
 
     @asyncio.coroutine
     def acquire(self):
-        raise asyncio.Return(True)
+        return(True)
 
     def release(self):
         pass
diff --git a/catkin_tools/execution/executor.py b/catkin_tools/execution/executor.py
index db04773..b3ad2a8 100644
--- a/catkin_tools/execution/executor.py
+++ b/catkin_tools/execution/executor.py
@@ -18,7 +18,7 @@ import traceback
 
 from itertools import tee
 
-import trollius as asyncio
+import asyncio
 
 from concurrent.futures import ThreadPoolExecutor
 from concurrent.futures import FIRST_COMPLETED
@@ -70,7 +70,7 @@ def async_job(verb, job, threadpool, locks, event_queue, log_path):
         # Check for stage synchronization lock
         if stage.locked_resource is not None:
             lock = locks.setdefault(stage.locked_resource, asyncio.Lock())
-            yield asyncio.From(lock)
+            yield from(lock)
         else:
             lock = FakeLock()
 
@@ -79,7 +79,7 @@ def async_job(verb, job, threadpool, locks, event_queue, log_path):
             if stage.occupy_job:
                 if not occupying_job:
                     while job_server.try_acquire() is None:
-                        yield asyncio.From(asyncio.sleep(0.05))
+                        yield from(asyncio.sleep(0.05))
                     occupying_job = True
             else:
                 if occupying_job:
@@ -103,7 +103,7 @@ def async_job(verb, job, threadpool, locks, event_queue, log_path):
                             # Get the logger
                             protocol_type = stage.logger_factory(verb, job.jid, stage.label, event_queue, log_path)
                             # Start asynchroonous execution
-                            transport, logger = yield asyncio.From(
+                            transport, logger = yield from(
                                 async_execute_process(
                                     protocol_type,
                                     **stage.async_execute_process_kwargs))
@@ -112,7 +112,7 @@ def async_job(verb, job, threadpool, locks, event_queue, log_path):
                             if 'Text file busy' in str(exc):
                                 # This is a transient error, try again shortly
                                 # TODO: report the file causing the problem (exc.filename)
-                                yield asyncio.From(asyncio.sleep(0.01))
+                                yield from(asyncio.sleep(0.01))
                                 continue
                             raise
 
@@ -125,7 +125,7 @@ def async_job(verb, job, threadpool, locks, event_queue, log_path):
                         **stage.async_execute_process_kwargs))
 
                     # Asynchronously yield until this command is completed
-                    retcode = yield asyncio.From(logger.complete)
+                    retcode = yield from(logger.complete)
                 except:  # noqa: E722
                     # Bare except is permissable here because the set of errors which the CommandState might raise
                     # is unbounded. We capture the traceback here and save it to the build's log files.
@@ -137,7 +137,7 @@ def async_job(verb, job, threadpool, locks, event_queue, log_path):
                 logger = IOBufferLogger(verb, job.jid, stage.label, event_queue, log_path)
                 try:
                     # Asynchronously yield until this function is completed
-                    retcode = yield asyncio.From(get_loop().run_in_executor(
+                    retcode = yield from(get_loop().run_in_executor(
                         threadpool,
                         stage.function,
                         logger,
@@ -179,7 +179,7 @@ def async_job(verb, job, threadpool, locks, event_queue, log_path):
             lock.release()
 
     # Finally, return whether all stages of the job completed
-    raise asyncio.Return(job.jid, all_stages_succeeded)
+    return(job.jid, all_stages_succeeded)
 
 
 @asyncio.coroutine
@@ -272,14 +272,14 @@ def execute_jobs(
         ))
 
         # Process jobs as they complete asynchronously
-        done_job_fs, active_job_fs = yield asyncio.From(asyncio.wait(
+        done_job_fs, active_job_fs = yield from(asyncio.wait(
             active_job_fs,
             timeout=0.10,
             return_when=FIRST_COMPLETED))
 
         for done_job_f in done_job_fs:
             # Capture a result once the job has finished
-            job_id, succeeded = yield asyncio.From(done_job_f)
+            job_id, succeeded = yield from(done_job_f)
 
             # Release a jobserver token now that this job has succeeded
             job_server.release(job_id)
@@ -362,7 +362,7 @@ def execute_jobs(
         completed=completed_jobs
     ))
 
-    raise asyncio.Return(all(completed_jobs.values()))
+    return(all(completed_jobs.values()))
 
 
 def run_until_complete(coroutine):
diff --git a/catkin_tools/verbs/catkin_build/build.py b/catkin_tools/verbs/catkin_build/build.py
index 5ed56ee..eb64988 100644
--- a/catkin_tools/verbs/catkin_build/build.py
+++ b/catkin_tools/verbs/catkin_build/build.py
@@ -22,7 +22,7 @@ import time
 import traceback
 import yaml
 
-import trollius as asyncio
+import asyncio
 
 try:
     # Python3
diff --git a/requirements.txt b/requirements.txt
index 184fceb..06ceec8 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -3,4 +3,3 @@ osrf_pycommon
 pyyaml
 setuptools
 sphinxcontrib-programoutput
-trollius
diff --git a/setup.py b/setup.py
index 03b728d..972ecae 100644
--- a/setup.py
+++ b/setup.py
@@ -15,10 +15,7 @@ install_requires = [
     'setuptools',
     'PyYAML',
     'osrf-pycommon > 0.1.1',
-    'trollius'
 ]
-if sys.version_info[0] == 2 and sys.version_info[1] <= 6:
-    install_requires.append('argparse')
 
 # Figure out the resources that need to be installed
 this_dir = os.path.abspath(os.path.dirname(__file__))


Test patch:
diff --git a/tests/README.md b/tests/README.md
index af24a4b..242adfb 100644
--- a/tests/README.md
+++ b/tests/README.md
@@ -14,7 +14,7 @@ All tests can be run from the root of the repository.
 
 First, make sure the required test dependencies are installed:
 
-*Ubuntu* -- `sudo apt-get install cmake libgtest-dev build-essential python-setuptools python-pip`
+*Ubuntu* -- `sudo apt-get install cmake libgtest-dev build-essential python3-setuptools python3-pip`
 
 *OS X* -- `pip install setuptools`
 
@@ -22,7 +22,7 @@ First, make sure the required test dependencies are installed:
 
 ```
 pip install argparse catkin-pkg distribute PyYAML psutil
-pip install nose coverage flake8 mock trollius empy --upgrade
+pip install nose coverage flake8 mock empy --upgrade
 pip install git+https://github.com/osrf/osrf_pycommon.git
 ```