Repository: ray-project/ray

Issue:
[tune] err_tb.format_exc() undefined in function_runner
### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Ubuntu 18.04
- **Ray installed from (source or binary)**: binary (PyPI)
- **Ray version**: 0.7.2
- **Python version**: 3.6
- **Exact command to reproduce**: N/A

### Describe the problem

When the objective function I am attempting to optimise with Tune crashes, Ray throws another exception complaining that the traceback object it gets has no `.format_exc()` attribute. It then claims that my function completed without dying or reporting any results.

### Source code / logs

Here is the relevant part of the traceback:

```
… traceback from my app up here…

(pid=21249) During handling of the above exception, another exception occurred:
(pid=21249)
(pid=21249) Traceback (most recent call last):
(pid=21249)   File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
(pid=21249)     self.run()
(pid=21249)   File "/home/sam/.virtualenvs/asnets/lib/python3.6/site-packages/ray/tune/function_runner.py", line 105, in run
(pid=21249)     err_tb = err_tb.format_exc()
(pid=21249) AttributeError: 'traceback' object has no attribute 'format_exc'
(pid=21249)
2019-07-07 10:14:16,574 ERROR trial_runner.py:487 -- Error processing event.
Traceback (most recent call last):
  File "/home/sam/.virtualenvs/asnets/lib/python3.6/site-packages/ray/tune/trial_runner.py", line 436, in _process_trial
    result = self.trial_executor.fetch_result(trial)
  File "/home/sam/.virtualenvs/asnets/lib/python3.6/site-packages/ray/tune/ray_trial_executor.py", line 323, in fetch_result
    result = ray.get(trial_future[0])
  File "/home/sam/.virtualenvs/asnets/lib/python3.6/site-packages/ray/worker.py", line 2195, in get
    raise value
ray.exceptions.RayTaskError: ray_WrappedFunc:train() (pid=21249, host=sam-ThinkPad-E470)
  File "/home/sam/.virtualenvs/asnets/lib/python3.6/site-packages/ray/tune/trainable.py", line 150, in train
    result = self._train()
  File "/home/sam/.virtualenvs/asnets/lib/python3.6/site-packages/ray/tune/function_runner.py", line 206, in _train
    ("Wrapped function ran until completion without reporting "
ray.tune.error.TuneError: Wrapped function ran until completion without reporting results or raising an exception.
```

The error seems to be due to [the following lines](https://github.com/ray-project/ray/blob/6a14f1a540d1c97d812cfcc2aecb1654028b279f/python/ray/tune/function_runner.py#L100-L108) in `function_runner.py`:

```python
# report the error but avoid indefinite blocking which would
# prevent the exception from being propagated in the unlikely
# case that something went terribly wrong
err_type, err_value, err_tb = sys.exc_info()
err_tb = err_tb.format_exc()
```

Python traceback objects don't seem to have a `format_exc()` attribute (at least as of 3.7), so I assume that last was a typo. Importing the `traceback` module and replacing `err_tb.format_exc()` with `traceback.format_tb(err_tb)` (or `traceback.format_exc()`) makes it go away.

[tune] err_tb.format_exc() undefined in function_runner
### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Ubuntu 18.04
- **Ray installed from (source or binary)**: binary (PyPI)
- **Ray version**: 0.7.2
- **Python version**: 3.6
- **Exact command to reproduce**: N/A

### Describe the problem

When the objective function I am attempting to optimise with Tune crashes, Ray throws another exception complaining that the traceback object it gets has no `.format_exc()` attribute. It then claims that my function completed without dying or reporting any results.

### Source code / logs

Here is the relevant part of the traceback:

```
… traceback from my app up here…

(pid=21249) During handling of the above exception, another exception occurred:
(pid=21249)
(pid=21249) Traceback (most recent call last):
(pid=21249)   File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
(pid=21249)     self.run()
(pid=21249)   File "/home/sam/.virtualenvs/asnets/lib/python3.6/site-packages/ray/tune/function_runner.py", line 105, in run
(pid=21249)     err_tb = err_tb.format_exc()
(pid=21249) AttributeError: 'traceback' object has no attribute 'format_exc'
(pid=21249)
2019-07-07 10:14:16,574 ERROR trial_runner.py:487 -- Error processing event.
Traceback (most recent call last):
  File "/home/sam/.virtualenvs/asnets/lib/python3.6/site-packages/ray/tune/trial_runner.py", line 436, in _process_trial
    result = self.trial_executor.fetch_result(trial)
  File "/home/sam/.virtualenvs/asnets/lib/python3.6/site-packages/ray/tune/ray_trial_executor.py", line 323, in fetch_result
    result = ray.get(trial_future[0])
  File "/home/sam/.virtualenvs/asnets/lib/python3.6/site-packages/ray/worker.py", line 2195, in get
    raise value
ray.exceptions.RayTaskError: ray_WrappedFunc:train() (pid=21249, host=sam-ThinkPad-E470)
  File "/home/sam/.virtualenvs/asnets/lib/python3.6/site-packages/ray/tune/trainable.py", line 150, in train
    result = self._train()
  File "/home/sam/.virtualenvs/asnets/lib/python3.6/site-packages/ray/tune/function_runner.py", line 206, in _train
    ("Wrapped function ran until completion without reporting "
ray.tune.error.TuneError: Wrapped function ran until completion without reporting results or raising an exception.
```

The error seems to be due to [the following lines](https://github.com/ray-project/ray/blob/6a14f1a540d1c97d812cfcc2aecb1654028b279f/python/ray/tune/function_runner.py#L100-L108) in `function_runner.py`:

```python
# report the error but avoid indefinite blocking which would
# prevent the exception from being propagated in the unlikely
# case that something went terribly wrong
err_type, err_value, err_tb = sys.exc_info()
err_tb = err_tb.format_exc()
```

Python traceback objects don't seem to have a `format_exc()` attribute (at least as of 3.7), so I assume that last was a typo. Importing the `traceback` module and replacing `err_tb.format_exc()` with `traceback.format_tb(err_tb)` (or `traceback.format_exc()`) makes it go away.



Discussion:

Resolving patch:
diff --git a/python/ray/tune/function_runner.py b/python/ray/tune/function_runner.py
--- a/python/ray/tune/function_runner.py
+++ b/python/ray/tune/function_runner.py
@@ -3,10 +3,10 @@
 from __future__ import print_function
 
 import logging
-import sys
 import time
 import inspect
 import threading
+import traceback
 from six.moves import queue
 
 from ray.tune import track
@@ -100,12 +100,9 @@ def run(self):
                 # report the error but avoid indefinite blocking which would
                 # prevent the exception from being propagated in the unlikely
                 # case that something went terribly wrong
-                err_type, err_value, err_tb = sys.exc_info()
-                err_tb = err_tb.format_exc()
+                err_tb_str = traceback.format_exc()
                 self._error_queue.put(
-                    (err_type, err_value, err_tb),
-                    block=True,
-                    timeout=ERROR_REPORT_TIMEOUT)
+                    err_tb_str, block=True, timeout=ERROR_REPORT_TIMEOUT)
             except queue.Full:
                 logger.critical(
                     ("Runner Thread was unable to report error to main "
@@ -234,13 +231,10 @@ def _stop(self):
 
     def _report_thread_runner_error(self, block=False):
         try:
-            err_type, err_value, err_tb = self._error_queue.get(
+            err_tb_str = self._error_queue.get(
                 block=block, timeout=ERROR_FETCH_TIMEOUT)
-            raise TuneError(("Trial raised a {err_type} exception with value: "
-                             "{err_value}\nWith traceback:\n{err_tb}").format(
-                                 err_type=err_type,
-                                 err_value=err_value,
-                                 err_tb=err_tb))
+            raise TuneError(("Trial raised an exception. Traceback:\n{}"
+                             .format(err_tb_str)))
         except queue.Empty:
             pass
 


Test patch: