Repository: ray-project/ray

Issue:
[tune] Feature request: tune.sample_from does not support callable objects. 
### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Linux Ubuntu 16.04
- **Ray installed from (source or binary)**: binary
- **Ray version**: 0.7.2
- **Python version**: 3.2
- **Exact command to reproduce**: See below

### Describe the problem

The `tune` sample_from interface is strictly limited to function objects, such as lambdas. This serves most use cases, but there are a number of instances where it's very useful to define a callable object to yield samples. (See trivial example below.) At the moment, providing a callable object returns errors from within tune variant generation, as the non-function-based `sample_from` entries are processed in grid entries. This can be resolved by changeing the sample/grid check from a direct check for `FunctionType` (Source location: https://github.com/ray-project/ray/blob/fadfa5f30bb654a74c781eaf8396a35af3ab7760/python/ray/tune/suggest/variant_generator.py#L116) to the builtin function `callable`.

I'm not entirely clear if this is an intentional limitation, and changing this logic will likely require expansion of tune's tests and documentation to cover the new behavior. I would be happy to open a PR for this if a maintainer gives the feature a 👍.

### Source code / logs

```python

import random

import ray.tune as tune
from ray.tune.suggest.variant_generator import generate_variants

class Normal:
    def __call__(self, _config):
        return random.normalvariate(mu=0, sigma=1)


grid_config = {"grid": tune.grid_search(list(range(2)))}
sample_config = {"normal": tune.sample_from(Normal())}


print(grid_config)
print(list(generate_variants(grid_config)))

print(sample_config)
print(list(generate_variants(sample_config)))

```

Results:

```
{'grid': {'grid_search': [0, 1]}}
[('grid=0', {'grid': 0}), ('grid=1', {'grid': 1})]
{'normal': tune.sample_from(<__main__.Normal object at 0x7f08ed1d0f50>)}
Traceback (most recent call last):
  File "sample_error.py", line 19, in <module>
    print(list(generate_variants(sample_config)))
  File "/work/home/lexaf/workspace/alphabeta/.conda/lib/python3.7/site-packages/ray/tune/suggest/variant_generator.py", line 43, in generate_variants
    for resolved_vars, spec in _generate_variants(unresolved_spec):
  File "/work/home/lexaf/workspace/alphabeta/.conda/lib/python3.7/site-packages/ray/tune/suggest/variant_generator.py", line 123, in _generate_variants
    for resolved_spec in grid_search:
  File "/work/home/lexaf/workspace/alphabeta/.conda/lib/python3.7/site-packages/ray/tune/suggest/variant_generator.py", line 193, in _grid_search_generator
    while value_indices[-1] < len(grid_vars[-1][1]):
TypeError: object of type 'Normal' has no len()
```



Discussion:
@asford this would be great! I'd be more than happy to help shepherd this.
I just had a case where this feature would've been useful.

Resolving patch:
diff --git a/python/ray/tune/suggest/variant_generator.py b/python/ray/tune/suggest/variant_generator.py
--- a/python/ray/tune/suggest/variant_generator.py
+++ b/python/ray/tune/suggest/variant_generator.py
@@ -2,7 +2,6 @@
 import logging
 import numpy
 import random
-import types
 
 from ray.tune import TuneError
 from ray.tune.sample import sample_from
@@ -126,7 +125,7 @@ def _generate_variants(spec):
     grid_vars = []
     lambda_vars = []
     for path, value in unresolved.items():
-        if isinstance(value, types.FunctionType):
+        if callable(value):
             lambda_vars.append((path, value))
         else:
             grid_vars.append((path, value))


Test patch:
diff --git a/python/ray/tune/tests/test_var.py b/python/ray/tune/tests/test_var.py
--- a/python/ray/tune/tests/test_var.py
+++ b/python/ray/tune/tests/test_var.py
@@ -1,5 +1,6 @@
 import os
 import numpy as np
+import random
 import unittest
 
 import ray
@@ -210,6 +211,29 @@ def testDependentGridSearch(self):
         self.assertEqual(trials[0].config, {"x": 100, "y": 1})
         self.assertEqual(trials[1].config, {"x": 200, "y": 1})
 
+    def testDependentGridSearchCallable(self):
+        class Normal:
+            def __call__(self, _config):
+                return random.normalvariate(mu=0, sigma=1)
+
+        class Single:
+            def __call__(self, _config):
+                return 20
+
+        trials = self.generate_trials({
+            "run": "PPO",
+            "config": {
+                "x": grid_search(
+                    [tune.sample_from(Normal()),
+                     tune.sample_from(Normal())]),
+                "y": tune.sample_from(Single()),
+            },
+        }, "dependent_grid_search")
+        trials = list(trials)
+        self.assertEqual(len(trials), 2)
+        self.assertEqual(trials[0].config["y"], 20)
+        self.assertEqual(trials[1].config["y"], 20)
+
     def testNestedValues(self):
         trials = self.generate_trials({
             "run": "PPO",