Trying xgboost inside forward , neural net

username: Bibhabasu_Mohapatra
cooked: <pre><code class="lang-auto">import torch
import torch.nn as nn
import timm
from xgboost import XGBClassifier
model_name = 'swin_base_patch4_window7_224'

out_dim    = 1


class get_model(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.model = timm.create_model(model_name, pretrained=False)
        self.model.head = nn.Sequential(nn.Linear(56,768),
                                              nn.Linear(768,256))
        self.last = nn.Linear(256, 128)
        self.depth1 = nn.Linear(128,64)
        self.depth2 = nn.Linear(64,6)
    def forward(self, features,targets):
        x = self.model.head(features)
        x = self.last(x)
        x = self.depth1(x)
        x = self.depth2(x)
        clf = XGBClassifier(tree_method='gpu_hist',
                             num_class = 6,objective = 'multi:softprob')
        clf.fit(x.detach().cpu().numpy(),targets.detach().cpu().numpy())
        preds = clf.predict(x.detach().cpu().numpy())
        preds = torch.tensor(preds)
        return preds
</code></pre>
<p>Everything is on GPU … Anything that maybe wrong was this model!!  Am i missing something?</p>

username: duyducvo4444
cooked: <p>The error log should show which line is producing the error, you can refer on that.<br>
Also, why detach in the middle of the forward function?</p>
<pre><code class="lang-auto">clf.fit(x.detach().cpu().numpy(),targets.detach().cpu().numpy())
preds = clf.predict(x.detach().cpu().numpy())
</code></pre>
<p>You know that the back-prop cannot continue from these two lines upward, right?</p>

username: Bibhabasu_Mohapatra
cooked: <p>yes! …  but didnt thought about that!   … do I have to custom the backprop?</p>

username: Bibhabasu_Mohapatra
cooked: <p><a class="mention" href="/u/duyducvo4444">@duyducvo4444</a>  actually I was trying last return type from XGB … dont think anybody has done it !!  yes backprop is problem!</p>

username: duyducvo4444
cooked: <p>I don’t think you can even customize your backprop because the graphs were already detached (i.e. it’s already lost, nothing remains). Backprop needs the graphs to calculate the gradients. That’s why I asked why did you detach in the middle of the forward function.<br>
I think the XGBClassifier you use here is not implemented on tensor, yes?</p>

username: Bibhabasu_Mohapatra
cooked: <p>yes… XGB  does not support tensor!</p>

username: Bibhabasu_Mohapatra
cooked: <p>I have to train …  and save the previous layer and return it with the preds and then IG it might work with xgb and saved last layer</p>

username: duyducvo4444
cooked: <p>From the look of it. I think you are trying to train your network to learn the best inputs for XGBClassifier from some features.<br>
But from this line</p>
<pre><code class="lang-auto">clf.fit(x.detach().cpu().numpy(),targets.detach().cpu().numpy())
</code></pre>
<p>Are you trying to make <em>x</em> to be the same as <em><strong>target</strong></em>?</p>

username: Bibhabasu_Mohapatra
cooked: <p>Yes… I was  trying to learn best inputs for xgb rather than given features and wanted out the outputs of xgb as final outputs…<br>
But  think its not possible that way.</p>

username: duyducvo4444
cooked: <p>If you are trying to make <em><strong>x</strong></em> to be the same as <em><strong>target</strong></em> ?<br>
Why don’t you just</p>
<pre><code class="lang-auto">def forward(self, features,targets):
        x = self.model.head(features)
        x = self.last(x)
        x = self.depth1(x)
        x = self.depth2(x)
        loss = some_loss_function(x, targets)
        return x, loss
</code></pre>
<p>Then back-prop using that loss and do the</p>
<pre><code class="lang-auto">clf.predict(x.detach().cpu().numpy())
</code></pre>
<p>after the training is done?</p>

username: Bibhabasu_Mohapatra
cooked: <p>great!!  thanks <a class="mention" href="/u/duyducvo4444">@duyducvo4444</a></p>