Torch.compile() InternalTorchDynamoError

whatisslove7: <p>Hello!<br>
I try to use <code>torch.complie()</code> function, but when testing this function with my train_epoch and elav_epoch, the following error appears:</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
File /opt/conda/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py:324, in _compile(code, globals, locals, builtins, compiler_fn, one_graph, export, hooks, frame)
    323 try:
--&gt; 324     out_code = transform_code_object(code, transform)
    325     orig_code_map[out_code] = code

File /opt/conda/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py:445, in transform_code_object(code, transformations, safe)
    443 propagate_line_nums(instructions)
--&gt; 445 transformations(instructions, code_options)
    446 return clean_and_assemble_instructions(instructions, keys, code_options)[1]

File /opt/conda/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py:299, in _compile.&lt;locals&gt;.transform(instructions, code_options)
    298 nonlocal output
--&gt; 299 tracer = InstructionTranslator(
    300     instructions,
    301     code,
    302     locals,
    303     globals,
    304     builtins,
    305     code_options,
    306     compiler_fn,
    307     one_graph,
    308     export,
    309     mutated_closure_cell_contents,
    310 )
    311 tracer.run()

File /opt/conda/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:1670, in InstructionTranslator.__init__(self, instructions, f_code, f_locals, f_globals, f_builtins, code_options, compiler_fn, one_graph, export, mutated_closure_cell_contents)
   1668 vars.extend(x for x in self.cell_and_freevars() if x not in vars)
-&gt; 1670 self.symbolic_locals = collections.OrderedDict(
   1671     (
   1672         k,
   1673         VariableBuilder(
   1674             self,
   1675             LocalInputSource(k, code_options["co_varnames"].index(k))
   1676             if k in code_options["co_varnames"]
   1677             else LocalSource((k)),
   1678         )(f_locals[k]),
   1679     )
   1680     for k in vars
   1681     if k in f_locals
   1682 )
   1684 # symbolic_locals contains the mapping from original f_locals to the
   1685 # Variable objects. During the Variable building phase, each object also
   1686 # has its associated guards. At the end, we will accumulate these
   (...)
   1699 # next invocation when args is not a list, and args[0] is a runtime
   1700 # error. Therefore, we recursively add guards for list/dict variable here.

File /opt/conda/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:1673, in &lt;genexpr&gt;(.0)
   1668 vars.extend(x for x in self.cell_and_freevars() if x not in vars)
   1670 self.symbolic_locals = collections.OrderedDict(
   1671     (
   1672         k,
-&gt; 1673         VariableBuilder(
   1674             self,
   1675             LocalInputSource(k, code_options["co_varnames"].index(k))
   1676             if k in code_options["co_varnames"]
   1677             else LocalSource((k)),
   1678         )(f_locals[k]),
   1679     )
   1680     for k in vars
   1681     if k in f_locals
   1682 )
   1684 # symbolic_locals contains the mapping from original f_locals to the
   1685 # Variable objects. During the Variable building phase, each object also
   1686 # has its associated guards. At the end, we will accumulate these
   (...)
   1699 # next invocation when args is not a list, and args[0] is a runtime
   1700 # error. Therefore, we recursively add guards for list/dict variable here.

File /opt/conda/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py:172, in VariableBuilder.__call__(self, value)
    171     return self.tx.output.side_effects[value]
--&gt; 172 return self._wrap(value).clone(**self.options())

File /opt/conda/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py:446, in VariableBuilder._wrap(self, value)
    437     return NumpyVariable(
    438         value,
    439         source=self.source,
   (...)
    444         ),
    445     )
--&gt; 446 elif value in tensor_dunder_fns:
    447     return TorchVariable(
    448         value,
    449         source=self.source,
    450         guards=make_guards(GuardBuilder.FUNCTION_MATCH),
    451     )

File /opt/conda/lib/python3.10/site-packages/tqdm/utils.py:75, in Comparable.__eq__(self, other)
     74 def __eq__(self, other):
---&gt; 75     return self._comparable == other._comparable

AttributeError: 'function' object has no attribute '_comparable'

Set torch._dynamo.config.verbose=True for more information


You can suppress this exception and fall back to eager by setting:
    torch._dynamo.config.suppress_errors = True


The above exception was the direct cause of the following exception:

InternalTorchDynamoError                  Traceback (most recent call last)
Cell In[47], line 6
      3 best_valid_loss = 1e12
      5 for epoch in range(CONFIG['epochs']):
----&gt; 6     train_loss = train_opt(model, optimizer, criterion, scheduler, train_dataloader)
      7     eval_loss = eval_opt(model, criterion, valid_dataloader)
      9     if valid_loss &lt; best_valid_loss:

File /opt/conda/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py:209, in _TorchDynamoContext.__call__.&lt;locals&gt;._fn(*args, **kwargs)
    207 dynamic_ctx.__enter__()
    208 try:
--&gt; 209     return fn(*args, **kwargs)
    210 finally:
    211     set_eval_frame(prior)

Cell In[34], line 4, in train_epoch(model, optimizer, criterion, scheduler, dataloader)
      3 def train_epoch(model, optimizer, criterion, scheduler, dataloader):
----&gt; 4     model.train()
      5     running_loss = 0.0
      7     for src, trg in tqdm(dataloader):

Cell In[34], line 7, in &lt;graph break in train_epoch&gt;(___stack0, model, optimizer, criterion, scheduler, dataloader)
      4 model.train()
      5 running_loss = 0.0
----&gt; 7 for src, trg in tqdm(dataloader):
      8     src = src.to(device)
      9     trg = trg.to(device) # (batch_size, seq_len)

File /opt/conda/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py:337, in catch_errors_wrapper.&lt;locals&gt;.catch_errors(frame, cache_size)
    334             return hijacked_callback(frame, cache_size, hooks)
    336 with compile_lock:
--&gt; 337     return callback(frame, cache_size, hooks)

File /opt/conda/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py:404, in convert_frame.&lt;locals&gt;._convert_frame(frame, cache_size, hooks)
    402 counters["frames"]["total"] += 1
    403 try:
--&gt; 404     result = inner_convert(frame, cache_size, hooks)
    405     counters["frames"]["ok"] += 1
    406     return result

File /opt/conda/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py:104, in wrap_convert_context.&lt;locals&gt;._fn(*args, **kwargs)
    102 torch.fx.graph_module._forward_from_src = fx_forward_from_src_skip_result
    103 try:
--&gt; 104     return fn(*args, **kwargs)
    105 finally:
    106     torch._C._set_grad_enabled(prior_grad_mode)

File /opt/conda/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py:262, in convert_frame_assert.&lt;locals&gt;._convert_frame_assert(frame, cache_size, hooks)
    259 global initial_grad_state
    260 initial_grad_state = torch.is_grad_enabled()
--&gt; 262 return _compile(
    263     frame.f_code,
    264     frame.f_globals,
    265     frame.f_locals,
    266     frame.f_builtins,
    267     compiler_fn,
    268     one_graph,
    269     export,
    270     hooks,
    271     frame,
    272 )

File /opt/conda/lib/python3.10/site-packages/torch/_dynamo/utils.py:163, in dynamo_timed.&lt;locals&gt;.dynamo_timed_inner.&lt;locals&gt;.time_wrapper(*args, **kwargs)
    161     compilation_metrics[key] = []
    162 t0 = time.time()
--&gt; 163 r = func(*args, **kwargs)
    164 time_spent = time.time() - t0
    165 # print(f"Dynamo timer: key={key}, latency={latency:.2f} sec")

File /opt/conda/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py:394, in _compile(code, globals, locals, builtins, compiler_fn, one_graph, export, hooks, frame)
    392 except Exception as e:
    393     exception_handler(e, code, frame)
--&gt; 394     raise InternalTorchDynamoError() from e

InternalTorchDynamoError: 
</code></pre>
<p>I use the following code:</p>
<pre><code class="lang-auto">train_opt = torch.compile(train_epoch, mode="reduce-overhead")
eval_opt = torch.compile(eval_epoch, mode="reduce-overhead")

#!g1.1
# num_epochs = 3
best_valid_loss = 1e12

for epoch in range(CONFIG['epochs']):
    train_opt = train_opt(model, optimizer, criterion, scheduler, train_dataloader)
    eval_opt = eval_opt(model, criterion, valid_dataloader)

    if valid_loss &lt; best_valid_loss:
        best_valid_loss = valid_loss
        torch.save(model.state_dict(), os.path.join(path, model_checkpoint))
        print(f"Model is saved to {os.path.join(path, model_checkpoint)}")

    print(f"Epoch №{epoch + 1}:")
    print(f"Training Loss: {train_loss}")
    print(f"Validation Loss: {valid_loss}")
    print()
</code></pre>
<p>Train and eval funcs:</p>
<pre><code class="lang-auto">#!g1.1
# add Train Loop
#!g1.1
# num_epochs = 3
best_valid_loss = 1e12

for epoch in range(CONFIG['epochs']):
    train_loss = train_epoch(model, optimizer, criterion, scheduler, train_dataloader)
    valid_loss = eval_epoch(model, criterion, valid_dataloader)

    if valid_loss &lt; best_valid_loss:
        best_valid_loss = valid_loss
        torch.save(model.state_dict(), os.path.join(path, model_checkpoint))
        print(f"Model is saved to {os.path.join(path, model_checkpoint)}")

    print(f"Epoch №{epoch + 1}:")
    print(f"Training Loss: {train_loss}")
    print(f"Validation Loss: {valid_loss}")
    print()
</code></pre>

ptrblck: <p>Could you add the missing code pieces to create a minimal and executable code snippet, which we could use to reproduce and debug the issue?</p>

whatisslove7: <p>I can open access to the notebook with all the code, will it be convenient for you?</p>

ptrblck: <p>That might work as long as no data dependencies are used, which might be tricky to get access to.<br>
Ideally, you would replace the inputs with random tensors as I would assume the actual data values are not needed to reproduce the error.</p>

whatisslove7: <p>I tried to substitute a tensor of the same dimension instead of train_dataloader, but with random data - the same error appeared</p>
<p>Could you guess what might be going wrong? Thanks a lot!</p>

ptrblck: <p>No, as I would still need to see the code.</p>

whatisslove7: <p>Minimal and executable code snippet has a lot of code (as for me), so I create a notebook with project<br>
The problem is that to run it on the wmt16 dataset, you need tokenizers that cannot be added in the project - I will provide them if you say</p>
<p>I hope it will be convenient for you to view the code from the link. If not, I will add the code here or send it where you say</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://colab.research.google.com/drive/1meq8NhUQAFv644xTjxr1KqNGTheL30V4?usp=sharing">
  <header class="source">
      <img src="https://ssl.gstatic.com/colaboratory-static/common/be2bcab5543e35ffad868f32935ea6a2/img/favicon.ico" class="site-icon" width="128" height="128">

      <a href="https://colab.research.google.com/drive/1meq8NhUQAFv644xTjxr1KqNGTheL30V4?usp=sharing" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>

  <article class="onebox-body">
    <img src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar" width="260" height="260">

<h3><a href="https://colab.research.google.com/drive/1meq8NhUQAFv644xTjxr1KqNGTheL30V4?usp=sharing" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>