Pytorch: Different Input within Stacking Ensemble & Apex library!

Sang_Dinh: <p>I shortly will describe my project:<br>
+2 input for 2 networks: image(3x256x256) &amp; 20 features(1 tensor of 20 number)<br>
+Stacking ensemble. <a href="https://discuss.pytorch.org/t/custom-ensemble-approach/52024/4">more</a><br>
This is my model:</p>
<pre><code class="lang-auto">class MetaMelanoma(nn.Module):
    def __init__(self,out_dim=9,n_meta_features=0,n_meta_dim=[512, 128]):
            super(MetaMelanoma,self).__init__()
            self.enet = timm.create_model('tf_efficientnet_b0_ns',pretrained=True)
            self.n_meta_features = n_meta_features
            self.dropouts = nn.ModuleList([
            nn.Dropout(0.5) for _ in range(5)])
            in_ch = self.enet.classifier.in_features
            self.meta = nn.Sequential(
                nn.Linear(n_meta_features, n_meta_dim[0]),
                nn.BatchNorm1d(n_meta_dim[0]),
                Swish_Module(),
                nn.Dropout(p=0.3),
                nn.Linear(n_meta_dim[0], n_meta_dim[1]),
                nn.BatchNorm1d(n_meta_dim[1]),
                Swish_Module(),
                nn.Linear(n_meta_dim[1],9)
            )
            self.myfc = nn.Linear(in_ch, out_dim)
            self.enet.classifier = nn.Identity()
    def extract(self, x):
        x = self.enet(x)
        return x
    def forward(self, x,x_meta):
        x = self.extract(x).squeeze(-1).squeeze(-1)
        if self.n_meta_features &gt; 0:
            x_meta = self.meta(x_meta)
            x = torch.cat((x, x_meta), dim=1)
        for i, dropout in enumerate(self.dropouts):
            if i == 0:
                out = self.myfc(dropout(x))
            else:
                out += self.myfc(dropout(x))
        out /= len(self.dropouts)

        return out
</code></pre>
<p>In order to train the network, I need Apex Nvidia library for optimizer and loss function <a href="https://nvidia.github.io/apex/" rel="noopener nofollow ugc">more</a><br>
Problem: To install Apex, we need old cuda version satisfying pytorch version. However, I am using a sever from my school, I can’t change cuda version. So anyone have any solution for this, maybe the same library or protocol?</p>

ptrblck: <p><code>apex.amp</code> is deprecated so use the native mixed-precision training utility via <code>torch.cuda.amp</code>. You can find the examples <a href="https://pytorch.org/docs/stable/notes/amp_examples.html#amp-examples">here</a>.</p>

Diaz65: <p>Thanks for sharing this info this is useful keep it up<a href="https://www.skylightpaycard.me/" rel="noopener nofollow ugc">.</a></p>

Sang_Dinh: <p>Thank you very much, that was the solution for me. I searched for someone who had the same model, but unfortunately, I am having some problems there with torch.cuda.amp. Could you suggest some projects using different inputs in same the model?</p>

ptrblck: <p>What kind of issues are you seeing using <code>torch.cuda.amp</code>?<br>
Since this util. is built into PyTorch, you don’t have to build any extensions manually using a locally installed CUDA toolkit as described in your first post.</p>

MyCenturaHealth: <p>torch.amp provides convenience methods for mixed precision, where some operations use the torch.float32 (float) datatype and other operations use lower precision floating point datatype (lower_precision_fp)<a href="https://mycenturahealth.site/" rel="noopener nofollow ugc">:</a> torch.float16 (half) or torch.bfloat16.</p>

GrantElliot: <p>Thanks a lot<a href="https://paybyplatema.site/" rel="noopener nofollow ugc">.</a> I will try and let here know how it went</p>

SamuelChukweze: <p>Did it worked for you? If yes, can you also help me in this regard<a href="https://ehallpass.us/login/" rel="noopener nofollow ugc">.</a> Thanks in advance.</p>

HarryKane: <p>I can help you in this regard<a href="https://mostlyboating.com/do-canoes-tip-over-easily/" rel="noopener nofollow ugc">,</a> let me know if you need my assistance. I would be more than happy to help you out.</p>