Using the types returned by ray.tune.sample

asafbenj: <p>Hi,<br>
I’m trying to adapt the code from the PyTorch tutorial “<a href="https://pytorch.org/tutorials/beginner/hyperparameter_tuning_tutorial.html" rel="noopener nofollow ugc">Hyper-parameter tuning with Ray Tune</a>”. I set the config variable like this:<br>
<code>config = {"batch_size" : tune.sample_from(lambda _: 2**np.random.randint(2, 9))}</code><br>
but when I use it in the training function:<br>
<code>n_samps = n_feats * config["batch_size"]</code><br>
I get a<br>
<code>TypeError: unsupported operand type(s) for *: 'int' and 'Function'</code><br>
Also if I use<br>
<code>config = {"lr" : tune.loguniform(1e-4, 1e-1)}</code><br>
then doing<br>
<code>2 * config["lr"] </code><br>
results in a<br>
<code>TypeError: unsupported operand type(s) for *: 'int' and 'Float'</code><br>
Indeed, <code>config["lr"]</code> is a ray.tune.sample.Float, and I don’t undrstand how to use it.<br>
Any help would be appreciated.<br>
Thanks!</p>

superunification: <p>Are you actually running the tuning algorithm from ray tune? Like <code>tune.run_experiments()</code> or <code>tune.run()</code>?</p>
<p>Because if you do that then these values will be sampled and replaced in your config dict with normal float / int values. Only when you try to use your configuration without going through tune will it contain these <code>ray.tune.sample.Float</code> types. If you want to do the latter anyway, just for debugging or whatnot, then call <code>.sample()</code> on the <code>ray.tune.sample.Float</code> and it’ll produce a usable value.</p>

asafbenj: <p>Thanks for the super-fast reply!<br>
I am indeed using <code>tune.run()</code>. Adding below the beginning of Ray Tune’s output. BTW, this specific run was on a Google Colab notebook connected to my Windows PC as localhost, but I also tried it on a colab hosted connection and got the same error.</p>
<p><code>2020-12-09 09:59:22,510 INFO services.py:1090 -- View the Ray dashboard at [http://127.0.0.1:8265](http://127.0.0.1:8265/) 2020-12-09 09:59:29,556 WARNING worker.py:1091 -- The dashboard on node PC01235821 failed with the following error: Traceback (most recent call last): File "F:\conda_parent\envs\pytorch\lib\site-packages\ray\dashboard\dashboard.py", line 956, in &lt;module&gt; dashboard = Dashboard( File "F:\conda_parent\envs\pytorch\lib\site-packages\ray\dashboard\dashboard.py", line 512, in __init__ build_dir = setup_static_dir(self.app) File "F:\conda_parent\envs\pytorch\lib\site-packages\ray\dashboard\dashboard.py", line 405, in setup_static_dir raise OSError( FileNotFoundError: [Errno 2] Dashboard build directory not found. If installing from source, please follow the additional steps required to build the dashboard(cd python/ray/dashboard/client &amp;&amp; npm ci &amp;&amp; npm run build): 'F:\\conda_parent\\envs\\pytorch\\lib\\site-packages\\ray\\dashboard\\client/build' 2020-12-09 10:01:41,960 ERROR syncer.py:63 -- Log sync requires rsync to be installed. 2020-12-09 10:02:53,042 WARNING worker.py:1091 -- Warning: The actor ImplicitFunc has size 245397724 when pickled. It will be stored in Redis, which could cause memory issues. This may mean that its definition uses a large array or other object. 2020-12-09 10:03:03,080 WARNING util.py:137 -- The </code>start_trial<code> operation took 115.11174321174622 seconds to complete, which may be a performance bottleneck.</code></p>
<pre><code class="lang-auto">== Status ==
Memory usage on this node: 25.5/63.9 GiB
Using AsyncHyperBand: num_stopped=0 Bracket: Iter 8.000: None | Iter 4.000: None | Iter 2.000: None | Iter 1.000: None
Resources requested: 1/20 CPUs, 1/1 GPUs, 0.0/24.51 GiB heap, 0.0/8.45 GiB objects
Result logdir: C:\Users\asafbe\ray_results\inner_2020-12-09_09-59-36
Number of trials: 1/10 (1 RUNNING)
</code></pre>
<pre><code class="lang-auto">|Trial name|status|loc|batch_size|conv_out1|dropout|kern_size|lin_out1|lr|n_conv_blocks|n_head|n_hid_ff|n_secs_per_seq|n_trans_layers|stride|
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|inner_9257a_00000|RUNNING||32|8|0.431199|5|16|0.00230475|1|1|4|4|2|1|

(pid=17072) 2020-12-09 10:03:33,796 ERROR function_runner.py:254 -- Runner Thread raised error. (pid=17072) Traceback (most recent call last): (pid=17072) File "F:\conda_parent\envs\pytorch\lib\site-packages\ray\tune\function_runner.py", line 248, in run (pid=17072) self._entrypoint() (pid=17072) File "F:\conda_parent\envs\pytorch\lib\site-packages\ray\tune\function_runner.py", line 315, in entrypoint (pid=17072) return self._trainable_func(self.config, self._status_reporter, (pid=17072) File "F:\conda_parent\envs\pytorch\lib\site-packages\ray\tune\function_runner.py", line 575, in _trainable_func (pid=17072) output = fn() (pid=17072) File "F:\conda_parent\envs\pytorch\lib\site-packages\ray\tune\function_runner.py", line 643, in inner (pid=17072) fn(config, **fn_kwargs) (pid=17072) File "&lt;ipython-input-88-08f6fd4506a8&gt;", line 2, in main_train_eval (pid=17072) File "&lt;ipython-input-77-855e742065de&gt;", line 5, in call_batchify (pid=17072) TypeError: unsupported operand type(s) for *: 'int' and 'Function' (pid=17072) Exception in thread Thread-2: (pid=17072) Traceback (most recent call last): (pid=17072) File "F:\conda_parent\envs\pytorch\lib\threading.py", line 932, in _bootstrap_inner (pid=17072) self.run() (pid=17072) File "F:\conda_parent\envs\pytorch\lib\site-packages\ray\tune\function_runner.py", line 267, in run (pid=17072) raise e (pid=17072) File "F:\conda_parent\envs\pytorch\lib\site-packages\ray\tune\function_runner.py", line 248, in run (pid=17072) self._entrypoint() (pid=17072) File "F:\conda_parent\envs\pytorch\lib\site-packages\ray\tune\function_runner.py", line 315, in entrypoint (pid=17072) return self._trainable_func(self.config, self._status_reporter, (pid=17072) File "F:\conda_parent\envs\pytorch\lib\site-packages\ray\tune\function_runner.py", line 575, in _trainable_func (pid=17072) output = fn() (pid=17072) File "F:\conda_parent\envs\pytorch\lib\site-packages\ray\tune\function_runner.py", line 643, in inner (pid=17072) fn(config, **fn_kwargs) (pid=17072) File "&lt;ipython-input-88-08f6fd4506a8&gt;", line 2, in main_train_eval (pid=17072) File "&lt;ipython-input-77-855e742065de&gt;", line 5, in call_batchify (pid=17072) TypeError: unsupported operand type(s) for *: 'int' and 'Function'
</code></pre>