Can't convert my pytorch model to ONNX

Sto: <p>I used this repo (github/com/Turoad/lanedet) to convert a pytorch model that use mobilenetv2 as backbone To ONNX but I didn’t succeeded.</p>
<p>i got a Runtime error that says:</p>
<blockquote>
<p>RuntimeError: Exporting the operator eye to ONNX opset version 12 is not supported. Please open a bug to request ONNX export support for the missing operator.</p>
</blockquote>
<p>it’s really disappointing, looking to the good result that this model gives and the quick performance that it provides,</p>
<p>is there any way that I can fix this bug? because I need to convert it to ONNX and then to TF lite model to use it in Android App I will provide the pretrained model that I have used and the way that I follow in converting…</p>
<p>Thank you so much for helping!</p>
<p>my colab notebook:</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://colab.research.google.com/drive/18udIh8tNJvti7jKmR4jRaRO-oYDgRmvA?usp=sharing">
  <header class="source">
      <img src="https://ssl.gstatic.com/colaboratory-static/common/27ed55654157d4fccaba00d81cbe557a/img/favicon.ico" class="site-icon" width="128" height="128">

      <a href="https://colab.research.google.com/drive/18udIh8tNJvti7jKmR4jRaRO-oYDgRmvA?usp=sharing" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>

  <article class="onebox-body">
    <img width="260" height="260" src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar">

<h3><a href="https://colab.research.google.com/drive/18udIh8tNJvti7jKmR4jRaRO-oYDgRmvA?usp=sharing" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>the pretrained model that I use:</p>
<aside class="onebox googledrive" data-onebox-src="https://drive.google.com/file/d/1o3-BgLIQesurIyDCKGliqbo2inUA5cPw/view?usp=sharing">
  <header class="source">

      <a href="https://drive.google.com/file/d/1o3-BgLIQesurIyDCKGliqbo2inUA5cPw/view?usp=sharing" target="_blank" rel="noopener nofollow ugc">drive.google.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://drive.google.com/file/d/1o3-BgLIQesurIyDCKGliqbo2inUA5cPw/view?usp=sharing" target="_blank" rel="noopener nofollow ugc"><span class="googledocs-onebox-logo g-drive-logo"></span></a>



<h3><a href="https://drive.google.com/file/d/1o3-BgLIQesurIyDCKGliqbo2inUA5cPw/view?usp=sharing" target="_blank" rel="noopener nofollow ugc">laneatt_mobilenetv2_culane.pth</a></h3>

<p>Google Drive file.</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


marksaroufim: <p>The error message is telling you that there’s an unsupported operator called <code>eye</code> so you have a few options</p>
<ol>
<li>Refactor your code to remove <code>eye</code>
</li>
<li>Open up a bug on <code>pytorch/pytorch</code> and tag <a class="mention" href="/u/garymm">@garymm</a>
</li>
<li>Implement the custom op yourself <a href="https://pytorch.org/docs/stable/onnx.html#custom-operators" class="inline-onebox" rel="noopener nofollow ugc">torch.onnx — PyTorch 1.11.0 documentation</a>
</li>
<li>Update to a newer opset which does have <code>eye</code> supported, see what’s supported here <a href="https://github.com/pytorch/pytorch/tree/master/torch/onnx" class="inline-onebox" rel="noopener nofollow ugc">pytorch/torch/onnx at master · pytorch/pytorch · GitHub</a>
</li>
</ol>
<p>The way to think about this is every PyTorch operation needs to be implemented by some opset in ONNX , so there’s always a few ops that will be missing that you need workaround</p>

david08170322: <p>Hello, I encountered some problems when transferring the model. The problems are as follows:</p>
<pre><code class="lang-auto">def convert_onnx():
    x = torch.randn((1, 3, 224, 224)).cuda()
    config = Configs()
    device = torch.device("cuda", index=0)
    net = Net(config, device).to(device)
    net.load_state_dict(
        torch.load("/workspace/code/TReS/Save_TReS_50/spaq_6_2022/sv/bestmodel_6_2022"))
    net.eval()
    path = "/workspace/code/TReS/Save_TReS_50/"

    torch.onnx.export(net,
                      x,
                      "noise.onnx",
                      export_params=True,
                      opset_version=13,
                      training=False,
                      do_constant_folding=False,
                      input_names=['input'],
                      output_names=['output'],
                      dynamic_axes={"input": {0: "batch_size"}, "output": {0: "batch_size"}})
    print("save model to %s" % path)
</code></pre>
<p><strong>errors:</strong><br>
/workspace/code/TReS/models.py:29: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can’t record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!<br>
out = F.conv2d(input, self.filter, stride=self.stride, padding=self.padding, groups=input.shape[1])<br>
Traceback (most recent call last):<br>
File “/workspace/code/TReS/save_onnx.py”, line 63, in <br>
convert_onnx()<br>
File “/workspace/code/TReS/save_onnx.py”, line 36, in convert_onnx<br>
dynamic_axes={“input”: {0: “batch_size”}, “output”: {0: “batch_size”}})<br>
File “/opt/conda/envs/tres/lib/python3.6/site-packages/torch/onnx/<strong>init</strong>.py”, line 276, in export<br>
custom_opsets, enable_onnx_checker, use_external_data_format)<br>
File “/opt/conda/envs/tres/lib/python3.6/site-packages/torch/onnx/utils.py”, line 94, in export<br>
use_external_data_format=use_external_data_format)<br>
File “/opt/conda/envs/tres/lib/python3.6/site-packages/torch/onnx/utils.py”, line 698, in _export<br>
dynamic_axes=dynamic_axes)<br>
File “/opt/conda/envs/tres/lib/python3.6/site-packages/torch/onnx/utils.py”, line 465, in _model_to_graph<br>
module=module)<br>
File “/opt/conda/envs/tres/lib/python3.6/site-packages/torch/onnx/utils.py”, line 206, in _optimize_graph<br>
graph = torch._C._jit_pass_onnx(graph, operator_export_type)<br>
File “/opt/conda/envs/tres/lib/python3.6/site-packages/torch/onnx/<strong>init</strong>.py”, line 309, in _run_symbolic_function<br>
return utils._run_symbolic_function(*args, **kwargs)<br>
File “/opt/conda/envs/tres/lib/python3.6/site-packages/torch/onnx/utils.py”, line 990, in _run_symbolic_function<br>
symbolic_fn = _find_symbolic_in_registry(domain, op_name, opset_version, operator_export_type)<br>
File “/opt/conda/envs/tres/lib/python3.6/site-packages/torch/onnx/utils.py”, line 947, in _find_symbolic_in_registry<br>
return sym_registry.get_registered_op(op_name, domain, opset_version)<br>
File “/opt/conda/envs/tres/lib/python3.6/site-packages/torch/onnx/symbolic_registry.py”, line 112, in get_registered_op<br>
raise RuntimeError(msg)<br>
RuntimeError: Exporting the operator broadcast_tensors to ONNX opset version 13 is not supported. Please feel free to request support or submit a pull request on PyTorch GitHub.</p>
<p>I tried to find the unsupported operator in my model, using this command, but did not find it, would like to ask everyone how to find this operator and replace it：</p>
<pre><code class="lang-auto">    with torch.autograd.profiler.profile(enabled=True, use_cuda=True, record_shapes=False,
                                         profile_memory=False) as prof:
        outputs = model(dump_input)
    print(prof.table())
    prof.export_chrome_trace('./resnet50_sampling_profile.json')
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/d/7d465013ca1734e517dc45202310cb480360ca20.png" data-download-href="https://discuss.pytorch.org/uploads/default/7d465013ca1734e517dc45202310cb480360ca20" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/d/7d465013ca1734e517dc45202310cb480360ca20_2_690x181.png" alt="image" data-base62-sha1="hSeklWB0NeIABRtsDrj8W8X63S0" width="690" height="181" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/d/7d465013ca1734e517dc45202310cb480360ca20_2_690x181.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/d/7d465013ca1734e517dc45202310cb480360ca20_2_1035x271.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/d/7d465013ca1734e517dc45202310cb480360ca20_2_1380x362.png 2x" data-dominant-color="EFE8F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2516×661 105 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
My version is :<br>
pytorch                   1.8.0           py3.6_cuda11.1_cudnn8.0.5_0    pytorch<br>
Model Structure:</p>
<pre><code class="lang-auto">Net(
  (L2pooling_l1): L2pooling()
  (L2pooling_l2): L2pooling()
  (L2pooling_l3): L2pooling()
  (L2pooling_l4): L2pooling()
  (model): ResNet(
    (conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
    (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (relu): ReLU(inplace=True)
    (maxpool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)
    (layer1): Sequential(
      (0): Bottleneck(
        (conv1): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv3): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (downsample): Sequential(
          (0): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
          (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
      )
      (1): Bottleneck(
        (conv1): Conv2d(256, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv3): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
      )
      (2): Bottleneck(
        (conv1): Conv2d(256, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv3): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
      )
    )
    (layer2): Sequential(
      (0): Bottleneck(
        (conv1): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv3): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (downsample): Sequential(
          (0): Conv2d(256, 512, kernel_size=(1, 1), stride=(2, 2), bias=False)
          (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
      )
      (1): Bottleneck(
        (conv1): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv3): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
      )
      (2): Bottleneck(
        (conv1): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv3): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
      )
      (3): Bottleneck(
        (conv1): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv3): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
      )
    )
    (layer3): Sequential(
      (0): Bottleneck(
        (conv1): Conv2d(512, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (downsample): Sequential(
          (0): Conv2d(512, 1024, kernel_size=(1, 1), stride=(2, 2), bias=False)
          (1): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
      )
      (1): Bottleneck(
        (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
      )
      (2): Bottleneck(
        (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
      )
      (3): Bottleneck(
        (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
      )
      (4): Bottleneck(
        (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
      )
      (5): Bottleneck(
        (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
      )
    )
    (layer4): Sequential(
      (0): Bottleneck(
        (conv1): Conv2d(1024, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv3): Conv2d(512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): BatchNorm2d(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
        (downsample): Sequential(
          (0): Conv2d(1024, 2048, kernel_size=(1, 1), stride=(2, 2), bias=False)
          (1): BatchNorm2d(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
      )
      (1): Bottleneck(
        (conv1): Conv2d(2048, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv3): Conv2d(512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): BatchNorm2d(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
      )
      (2): Bottleneck(
        (conv1): Conv2d(2048, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (conv3): Conv2d(512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): BatchNorm2d(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu): ReLU(inplace=True)
      )
    )
    (avgpool): AdaptiveAvgPool2d(output_size=(1, 1))
    (fc): Linear(in_features=2048, out_features=1000, bias=True)
  )
  (transformer): Transformer(
    (encoder): TransformerEncoder(
      (layers): ModuleList(
        (0): TransformerEncoderLayer(
          (self_attn): MultiheadAttention(
            (out_proj): _LinearWithBias(in_features=3840, out_features=3840, bias=True)
          )
          (linear1): Linear(in_features=3840, out_features=64, bias=True)
          (dropout): Dropout(p=0.5, inplace=False)
          (linear2): Linear(in_features=64, out_features=3840, bias=True)
          (norm1): LayerNorm((3840,), eps=1e-05, elementwise_affine=True)
          (norm2): LayerNorm((3840,), eps=1e-05, elementwise_affine=True)
          (dropout1): Dropout(p=0.5, inplace=False)
          (dropout2): Dropout(p=0.5, inplace=False)
        )
        (1): TransformerEncoderLayer(
          (self_attn): MultiheadAttention(
            (out_proj): _LinearWithBias(in_features=3840, out_features=3840, bias=True)
          )
          (linear1): Linear(in_features=3840, out_features=64, bias=True)
          (dropout): Dropout(p=0.5, inplace=False)
          (linear2): Linear(in_features=64, out_features=3840, bias=True)
          (norm1): LayerNorm((3840,), eps=1e-05, elementwise_affine=True)
          (norm2): LayerNorm((3840,), eps=1e-05, elementwise_affine=True)
          (dropout1): Dropout(p=0.5, inplace=False)
          (dropout2): Dropout(p=0.5, inplace=False)
        )
      )
      (norm): LayerNorm((3840,), eps=1e-05, elementwise_affine=True)
    )
  )
  (position_embedding): PositionEmbeddingSine()
  (fc2): Linear(in_features=3840, out_features=2048, bias=True)
  (fc): Linear(in_features=4096, out_features=1, bias=True)
  (ReLU): ReLU()
  (avg7): AvgPool2d(kernel_size=(7, 7), stride=(7, 7), padding=0)
  (avg8): AvgPool2d(kernel_size=(8, 8), stride=(8, 8), padding=0)
  (avg4): AvgPool2d(kernel_size=(4, 4), stride=(4, 4), padding=0)
  (avg2): AvgPool2d(kernel_size=(2, 2), stride=(2, 2), padding=0)
  (drop2d): Dropout(p=0.1, inplace=False)
  (consistency): L1Loss()
)
</code></pre>
<p>maybe L1 loss not support?</p>
<pre><code class="lang-auto">consistloss1 = self.consistency(out_t_c, fout_t_c.detach())
consistloss2 = self.consistency(layer4, flayer4.detach())
consistloss = 1 * (consistloss1 + consistloss2)
</code></pre>
<ul>
<li>List item</li>
</ul>