Ray tune and ImplicitFunc is very large error

username: ko7129
cooked: <p>Greetings to the community!!</p>
<p>I am trying to grid search some parameters of my training function using ray tune.<br>
The input data to train_cifar() used for training and testing are 2 lists of dimensions<br>
400x13000 and 40x13000, respectively.</p>
<p>Due to size I cannot produce a reproducible example, but below I show three different<br>
ways I have tried to ray tune my model.</p>
<p>In each case I receive the following error:</p>
<p>The actor ImplicitFunc is very large (95 MiB). Check that its definition is not implicitly<br>
capturing a large array or other object in scope. Tip: use ray.put() to put large objects<br>
in the Ray object store.</p>
<p>or this one:</p>
<p>debug_error_string = “{“created”:”<span class="mention">@1643300850.335447653</span>",“description”:<br>
“Error received from peer ipv4:172.28.0.2:45437”, “file”:“src/core/lib/surface/call.cc”, “file_line”:1074, “grpc_message”: “Received message larger than max (137418486 vs. 104857600)”,“grpc_status”:8}"</p>
<p>I don’t understand what the limit of 95 MiB is since my lists are really small.</p>
<p>Any ideas of what am I doing wrong?</p>
<p>I am running the following codes to google’s Colab.</p>
<p>Kostas</p>
<p><strong>CODE I</strong></p>
<pre><code class="lang-auto">X_scaled_train = ...
X_scaled_test = ...

ray.init()
X_scaled_train1 = ray.put(X_scaled_train)
X_scaled_test1 = ray.put(X_scaled_test)

def train_cifar(config, data = None, checkpoint_dir=None):
  X_scaled_train_tmp = config["data1"]
  X_scaled_train2 = ray.get(X_scaled_train_tmp)

  X_scaled_test_tmp = config["data2"]
  X_scaled_test2 = ray.get(X_scaled_test_tmp)

def tunerTrain():
      config = {
        "data1" : X_scaled_train1,
        "data2" : X_scaled_test1,        
      }          
      scheduler = ASHAScheduler(
              ...
          )
      reporter = CLIReporter(
              ...
          )
      result = tune.run(
              partial(train_cifar, data_dir=data_dir),  
              ...
          )

tunerTrain()
</code></pre>
<p><strong>CODE II</strong></p>
<pre><code class="lang-auto">X_scaled_train = ...
X_scaled_test = ...

ray.init()
X_scaled_train1 = ray.put(X_scaled_train)
X_scaled_test1 = ray.put(X_scaled_test)

def train_cifar(config, data = None, checkpoint_dir=None):
  
  X_scaled_train2 = ray.get(data[0])
  X_scaled_test2 = ray.get(data[2])

def tunerTrain():
      config = {
              ...        
      }          
      scheduler = ASHAScheduler(
              ...
          )
      reporter = CLIReporter(
              ...
          )
      result = tune.run(
               tune.with_parameters(train_cifar, data=[X_scaled_train1, X_scaled_train_trait, 
                                                       X_scaled_test1, X_scaled_test_trait]),
              ...
          )

tunerTrain()
</code></pre>
<p><strong>CODE III</strong></p>
<pre><code class="lang-auto">X_scaled_train = ...
X_scaled_test = ...

def train_cifar(config, data = None, checkpoint_dir=None):
  
  X_scaled_train2 = data[0]
  X_scaled_test2 = data[2]

def tunerTrain():
      config = {
              ...        
      }          
      scheduler = ASHAScheduler(
              ...
          )
      reporter = CLIReporter(
              ...
          )
      result = tune.run(
               tune.with_parameters(train_cifar, data=[X_scaled_train, X_scaled_train_trait, 
                                                       X_scaled_test, X_scaled_test_trait]),
              ...
          )

tunerTrain()

</code></pre>

username: H-Huang
cooked: <p>This forum section is meant for the PyTorch distributed questions. It seems the code and error you are interested in is coming from Ray, so it would be best to ask on that forum (<a href="https://discuss.ray.io/" rel="noopener nofollow ugc">https://discuss.ray.io/</a>)</p>