Repository: ray-project/ray

Issue:
[tune] AssertionError: Resource invalid
<!--
General questions should be asked on the mailing list ray-dev@googlegroups.com.
Questions about how to use Ray should be asked on
[StackOverflow](https://stackoverflow.com/questions/tagged/ray).

Before submitting an issue, please fill out the following form.
-->

### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**:
Ubuntu 16.04
- **Ray installed from (source or binary)**:
pip install https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-0.8.0.dev4-cp36-cp36m-manylinux1_x86_64.whl
- **Ray version**:
0.8.0.dev4
- **Python version**:
3.6.7
- **Exact command to reproduce**:

<!--
You can obtain the Ray version with

python -c "import ray; print(ray.__version__)"
-->

### Describe the problem
<!-- Describe the problem clearly here. -->

I run 5 trials with ray.tune. In one of the trials (each time), an error occurs at the end of training: `AssertionError: Resource invalid: Resources(cpu=3, gpu=0.33, memory=0, object_store_memory=0, extra_cpu=0, extra_gpu=0, extra_memory=0, extra_object_store_memory=0, custom_resources={}, extra_custom_resources={})`.

When I trace back the error, I end up in the following function (ray/tune/resources.py):

```
def is_nonnegative(self):
    all_values = [self.cpu, self.gpu, self.extra_cpu, self.extra_gpu]
    all_values += list(self.custom_resources.values())
    all_values += list(self.extra_custom_resources.values())
    return all(v >= 0 for v in all_values)
```

It seems `custom_resources` and `extra_custom_resources` are not defined. It is weird that the error only occurs in one run... Is this a bug, or any suggestions on how to fix?


### Source code / logs
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->

__This is how I call `tune.run`__

```
tune.run(
    ModelTrainerMT,
    resources_per_trial={
        'cpu': config['ncpu'],
        'gpu': config['ngpu'],
    },
    num_samples=1,
    config=best_config,
    local_dir=store,
    raise_on_failed_trial=True,
    verbose=1,
    with_server=False,
    ray_auto_init=False,
    scheduler=early_stopping_scheduler,
    loggers=[JsonLogger, CSVLogger],
    checkpoint_at_end=True,
    reuse_actors=True,
    stop={'epoch': 2 if args.test else config['max_t']}
)
```

__Traceback__

```
2019-09-06 09:56:45,526 ERROR trial_runner.py:557 -- Error processing event.
Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/site-packages/ray/tune/trial_runner.py", line 552, in _process_trial
    self.trial_executor.stop_trial(trial)
  File "/opt/conda/lib/python3.6/site-packages/ray/tune/ray_trial_executor.py", line 246, in stop_trial
    self._return_resources(trial.resources)
  File "/opt/conda/lib/python3.6/site-packages/ray/tune/ray_trial_executor.py", line 388, in _return_resources
    "Resource invalid: {}".format(resources))
AssertionError: Resource invalid: Resources(cpu=3, gpu=0.33, memory=0, object_store_memory=0, extra_cpu=0, extra_gpu=0, extra_memory=0, extra_object_store_memory=0, custom_resources={}, extra_custom_resources={})
```





Discussion:
@richardliaw 

Resolving patch:
diff --git a/python/ray/tune/resources.py b/python/ray/tune/resources.py
--- a/python/ray/tune/resources.py
+++ b/python/ray/tune/resources.py
@@ -5,11 +5,10 @@
 from collections import namedtuple
 import logging
 import json
+from numbers import Number
 # For compatibility under py2 to consider unicode as str
 from six import string_types
 
-from numbers import Number
-
 from ray.tune import TuneError
 
 logger = logging.getLogger(__name__)
@@ -66,6 +65,23 @@ def __new__(cls,
             custom_resources.setdefault(value, 0)
             extra_custom_resources.setdefault(value, 0)
 
+        cpu = round(cpu, 2)
+        gpu = round(gpu, 2)
+        memory = round(memory, 2)
+        object_store_memory = round(object_store_memory, 2)
+        extra_cpu = round(extra_cpu, 2)
+        extra_gpu = round(extra_gpu, 2)
+        extra_memory = round(extra_memory, 2)
+        extra_object_store_memory = round(extra_object_store_memory, 2)
+        custom_resources = {
+            resource: round(value, 2)
+            for resource, value in custom_resources.items()
+        }
+        extra_custom_resources = {
+            resource: round(value, 2)
+            for resource, value in extra_custom_resources.items()
+        }
+
         all_values = [
             cpu, gpu, memory, object_store_memory, extra_cpu, extra_gpu,
             extra_memory, extra_object_store_memory


Test patch:
diff --git a/python/ray/tune/tests/test_trial_runner.py b/python/ray/tune/tests/test_trial_runner.py
--- a/python/ray/tune/tests/test_trial_runner.py
+++ b/python/ray/tune/tests/test_trial_runner.py
@@ -1531,6 +1531,14 @@ def testFractionalGpus(self):
         self.assertEqual(trials[2].status, Trial.PENDING)
         self.assertEqual(trials[3].status, Trial.PENDING)
 
+    def testResourceNumericalError(self):
+        resource = Resources(cpu=0.99, gpu=0.99, custom_resources={"a": 0.99})
+        small_resource = Resources(
+            cpu=0.33, gpu=0.33, custom_resources={"a": 0.33})
+        for i in range(3):
+            resource = Resources.subtract(resource, small_resource)
+        self.assertTrue(resource.is_nonnegative())
+
     def testResourceScheduler(self):
         ray.init(num_cpus=4, num_gpus=1)
         runner = TrialRunner()