Learning stagnation on sparse inputs

kirk86: <p>I’m trying to fit an MLP on tabular sparse data but from monitoring the loss it seems that there’s no learning happening</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/f/5f3aca30c45e5554d2255d1f73f47e71c48e26d9.png" data-download-href="https://discuss.pytorch.org/uploads/default/5f3aca30c45e5554d2255d1f73f47e71c48e26d9" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/f/5f3aca30c45e5554d2255d1f73f47e71c48e26d9_2_400x375.png" alt="image" data-base62-sha1="dArjBKA0DOlLnBVjbhSpHGJUk1X" width="400" height="375" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/f/5f3aca30c45e5554d2255d1f73f47e71c48e26d9_2_400x375.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/f/5f3aca30c45e5554d2255d1f73f47e71c48e26d9_2_600x562.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/f/5f3aca30c45e5554d2255d1f73f47e71c48e26d9_2_800x750.png 2x" data-dominant-color="DFDFDF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1248×1168 189 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Any ideas what might be going wrong?</p>

Andrei_Cristea: <p>Could you share more of your code, ideally a minimal reproducible snippet that has this issue? There could be a lot of different reasons why this is happening and without seeing some code it’s hard to diagnose it.</p>

kirk86: <p>Sure thing,</p>
<p>the code for loading data is taken from this <a href="https://discuss.pytorch.org/t/dataloader-loads-data-very-slow-on-sparse-tensor/117391/2">thread</a>.</p>
<p>What I can’t provide is the actual dataset.</p>
<p>Here’s an example:</p>
<pre><code class="lang-auto">class SparseDataset(tud.Dataset):
    """
    Custom Dataset class for scipy sparse matrix
    """
    def __init__(self, data:Union[np.ndarray, sps.coo_matrix, sps.csr_matrix],
                 targets:Union[np.ndarray, sps.coo_matrix, sps.csr_matrix], 
                 transform:bool = None):
        
        # Transform data coo_matrix to csr_matrix for indexing
        if type(data) == sps.coo_matrix:
            self.data = data.tocsr()
        else:
            self.data = data
            
        # Transform targets coo_matrix to csr_matrix for indexing
        if type(targets) == sps.coo_matrix:
            self.targets = targets.tocsr()
        else:
            self.targets = targets
        
        self.transform = transform # Can be removed

    def __getitem__(self, index:int):
        return self.data[index], self.targets[index]

    def __len__(self):
        return self.data.shape[0]

def sparse_coo_to_tensor(coo:sps.coo_matrix):
    """
    Transform scipy coo matrix to pytorch sparse tensor
    """
    values = coo.data
    indices = np.vstack((coo.row, coo.col))
    shape = coo.shape

    i = torch.LongTensor(indices)
    v = torch.FloatTensor(values)
    s = torch.Size(shape)

    return torch.sparse.FloatTensor(i, v, s)

def sparse_batch_collate(batch:list): 
    """
    Collate function which to transform scipy coo matrix to pytorch sparse tensor
    """
    data_batch, targets_batch = zip(*batch)
    if type(data_batch[0]) == sps.csr_matrix:
        data_batch = sps.vstack(data_batch).tocoo()
        data_batch = sparse_coo_to_tensor(data_batch)
    else:
        data_batch = torch.FloatTensor(data_batch)

    if type(targets_batch[0]) == sps.csr_matrix:
        targets_batch = sps.vstack(targets_batch).tocoo()
        targets_batch = sparse_coo_to_tensor(targets_batch)
    else:
        targets_batch = torch.FloatTensor(targets_batch)
    return data_batch, targets_batch


class MLP(nn.Module):
    def __init__(self, in_dim, num_classes=2):
        super(LogisticRegression, self).__init__()
        
        self.in_dim = in_dim 
        self.num_classes = num_classes
        
        self.l1 = nn.Linear(in_dim, 2048) 
        self.d1 = nn.Dropout(p=0.5)
        self.l2 = nn.Linear(2048, 1024)
        self.l3 = nn.Linear(1024, 512)
        self.m1 = nn.MaxPool1d(kernel_size=2, stride=2)
        self.l4 = nn.Linear(256, 128)
        self.l5 = nn.Linear(128, 64)
        self.m2 = nn.MaxPool1d(kernel_size=2, stride=2)
        self.l6 = nn.Linear(32, 16)
        self.l7 = nn.Linear(16, 8)
        self.l8 = nn.Linear(8, num_classes)
    
    def forward(self, x):
        x = self.l1(x)
        x = self.d1(x)
        x = self.l2(x).relu_()
        x = self.l3(x)
        x = self.m1(x.unsqueeze(dim=1))
        x = self.l4(x.squeeze())
        x = self.l5(x)
        x = self.m2(x.unsqueeze(dim=1))
        x = self.l6(x.squeeze())
        x = self.l7(x).relu_()
        x = self.l8(x)
        return x

def criterion(model, x, y):
    y_hat = model(x)
    prob = torch.sigmoid(y_hat)[:, 1]
    return nn.BCELoss(reduction='mean')(prob, y), y_hat

X = sps.random(80000, 4096, density=0.25)
y = np.arange(80000)

dset = SparseDataset(X, y)

loader = tud.DataLoader(dset, batch_size=512, collate_fn=sparse_batch_collate)

Training proceed as usual......
</code></pre>

Andrei_Cristea: <p>Where is <code>sps</code> defined?</p>

kirk86: <p>sorry that’s just <code>from scipy import sparse as sps</code> and <code>tud</code> stands for <code>torch.utils.data</code></p>

Andrei_Cristea: <p>Could you share a stripped down / minimal version of your training loop too, please?</p>

kirk86: <pre><code class="lang-auto"> for i, (x, y) in enumerate(loader):

        if x.is_sparse:
            x = x.to_dense()
        
        x = x.cuda(non_blocking=True)
        y = y.cuda(non_blocking=True)

        loss, output = criterion(model, x, y)

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
</code></pre>

Andrei_Cristea: <p>Where do you define <code>LogisticRegression</code>?</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
~\AppData\Local\Temp\ipykernel_19676\3241591868.py in &lt;module&gt;
----&gt; 1 model = MLP(in_dim=409)
      2 for i, (x, y) in enumerate(loader):
      3     if x.is_sparse:
      4         x = x.to_dense()
      5 

~\AppData\Local\Temp\ipykernel_19676\2325333031.py in __init__(self, in_dim, num_classes)
     75 class MLP(nn.Module):
     76     def __init__(self, in_dim, num_classes=2):
---&gt; 77         super(LogisticRegression, self).__init__()
     78 
     79         self.in_dim = in_dim

NameError: name 'LogisticRegression' is not defined
</code></pre>
<p>Btw in code I’ve seen, the class inside the super() call is typically the parent class (in your case MLP) not some other one. Not sure if that makes any difference though.</p>

kirk86: <p>Simply change line77 from <code>super(LogisticRegression, self).__init__()</code> to <code>super(MLP, self).__init__()</code>.</p>
<p>That’s reminiscent of the old notation.</p>

Andrei_Cristea: <p>Getting this error now:</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
~\AppData\Local\Temp\ipykernel_21892\1132068329.py in &lt;module&gt;
      7     y = y.cuda(non_blocking=True)
      8 
----&gt; 9     loss, output = criterion(model, x, y)
     10 
     11     optimizer.zero_grad()

~\AppData\Local\Temp\ipykernel_21892\238189117.py in criterion(model, x, y)
    109     y_hat = model(x)
    110     prob = torch.sigmoid(y_hat)[:, 1]
--&gt; 111     return nn.BCELoss(reduction='mean')(prob, y), y_hat
    112 
    113 class SparseDataset(Dataset):

c:\users\andre\appdata\local\continuum\anaconda3\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--&gt; 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

c:\users\andre\appdata\local\continuum\anaconda3\lib\site-packages\torch\nn\modules\loss.py in forward(self, input, target)
    611     def forward(self, input: Tensor, target: Tensor) -&gt; Tensor:
    612         assert self.weight is None or isinstance(self.weight, Tensor)
--&gt; 613         return F.binary_cross_entropy(input, target, weight=self.weight, reduction=self.reduction)
    614 
    615 

c:\users\andre\appdata\local\continuum\anaconda3\lib\site-packages\torch\nn\functional.py in binary_cross_entropy(input, target, weight, size_average, reduce, reduction)
   2753         raise ValueError(
   2754             "Using a target size ({}) that is different to the input size ({}) is deprecated. "
-&gt; 2755             "Please ensure they have the same size.".format(target.size(), input.size())
   2756         )
   2757 

ValueError: Using a target size (torch.Size([16, 409])) that is different to the input size (torch.Size([16])) is deprecated. Please ensure they have the same size.
</code></pre>
<p>What’s your torch version? I have <code>1.8.1+cu111</code>.</p>

kirk86: <p>I don’t think there’s any relation to torch version, simply check the sizes of arguments to <code>BCELoss</code> in line <code>111</code>.</p>
<p>If for instance <code>prob.shape = (16, 1)</code> and <code>y.shape = (16,)</code> then <code>BCELoss</code> will complain simply flatten the <code>prob</code> argument in the loss function.</p>
<p>In other words both <code>prob</code> and <code>y</code> arguments in <code>BCELoss</code> should have same shapes.</p>

Andrei_Cristea: <aside class="quote no-group quote-modified" data-username="Andrei_Cristea" data-post="10" data-topic="153380">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/andrei_cristea/48/47843_2.png" class="avatar"> Andrei_Cristea:</div>
<blockquote>
<pre><code class="lang-auto">ValueError: Using a target size (torch.Size([512, 4096])) that is different to the input size (torch.Size([512])) is deprecated. Please ensure they have the same size.
</code></pre>
</blockquote>
</aside>
<p>Right, but using your code the sizes come out to (512, 4096) and (512), per the message above. This suggests it’s not simply a question of flattening, so I’m asking you to look into why there’s that size mismatch.</p>
<p>(note in my original post it was (16, 409) because I’d used a different batch size and matrix size, but the error remains with your original variables)</p>

kirk86: <p>No error on my side, same code with same modifications as suggested.</p>
<p>Her’s the output using batch size 512:</p>
<pre><code class="lang-auto">shapes, x torch.Size([512]), y torch.Size([512])
Loss : -25450.0
shapes, x torch.Size([512]), y torch.Size([512])
Loss : -76650.0
shapes, x torch.Size([512]), y torch.Size([512])
Loss : -127850.0
shapes, x torch.Size([512]), y torch.Size([512])
Loss : -179050.0
shapes, x torch.Size([512]), y torch.Size([512])
Loss : -230250.0
shapes, x torch.Size([512]), y torch.Size([512])
Loss : -281450.0
shapes, x torch.Size([512]), y torch.Size([512])
Loss : -332650.0
shapes, x torch.Size([512]), y torch.Size([512])
Loss : -383850.0
shapes, x torch.Size([512]), y torch.Size([512])
Loss : -435050.0
shapes, x torch.Size([512]), y torch.Size([512])
Loss : -486250.0
shapes, x torch.Size([512]), y torch.Size([512])
Loss : -537450.0
shapes, x torch.Size([512]), y torch.Size([512])
Loss : -588650.0
shapes, x torch.Size([512]), y torch.Size([512])
Loss : -639850.0
shapes, x torch.Size([512]), y torch.Size([512])
Loss : -691050.0
</code></pre>
<p>The shapes are for the arguments in <code>BCELoss</code>.</p>
<p>Here’s with batch size 16:</p>
<pre><code class="lang-auto">shapes, x = torch.Size([16]), y = torch.Size([16])
Loss : -650.0
shapes, x = torch.Size([16]), y = torch.Size([16])
Loss : -2250.0
shapes, x = torch.Size([16]), y = torch.Size([16])
Loss : -3850.0
shapes, x = torch.Size([16]), y = torch.Size([16])
Loss : -5450.0
shapes, x = torch.Size([16]), y = torch.Size([16])
Loss : -7050.0
shapes, x = torch.Size([16]), y = torch.Size([16])
Loss : -8650.0
shapes, x = torch.Size([16]), y = torch.Size([16])
Loss : -10250.0
shapes, x = torch.Size([16]), y = torch.Size([16])
Loss : -11850.0
shapes, x = torch.Size([16]), y = torch.Size([16])
Loss : -13450.0
shapes, x = torch.Size([16]), y = torch.Size([16])
Loss : -15050.0
shapes, x = torch.Size([16]), y = torch.Size([16])
Loss : -16650.0
</code></pre>
<p>And here’s my <code>torch</code> version <code>py3.7_cuda11.3_cudnn8.2.0_0    pytorch 1.11.0</code></p>
<p>I think there’s sth on your side causing that error. Most probably older version of <code>BCELoss</code>?</p>
<p>Are you sure you are calling the <code>criterion</code> as in the example <code>loss, output = criterion(model, x, y)</code> and the criterion should return back two arguments?</p>

Andrei_Cristea: <p>Hi, are you sure you don’t have any local changes to your code that aren’t reflected in the code you shared above?</p>
<p>I ran the code in a <a href="https://colab.research.google.com/drive/1Vd4VvbbLDIEHx5DHmknMgYiUKfGjWHeX?usp=sharing" rel="noopener nofollow ugc">colab</a> that you can access too and had the same issue as in my local environment. Note that I decreased batch size from 80k to 8k just to test this.</p>

kirk86: <p>That’s great thanks for the colab notebook. Now I can see where the error is.<br>
There’s plenty of redundancy in the colab notebook but that’s fine the error is due to the fact that <em>you’ve changed the size of the targets vector.</em></p>
<p>If you look at my initial code the targets are defined as a vector of similar length to the actual input.</p>
<p><code>y = np.arange(80000)</code></p>
<p>but in your code you have changed it to</p>
<p><code>y = (X &gt; 0.5).astype(int)</code></p>
<p>the shape of <code>y</code> is <code>y.shape = [8000, 4096]</code>.</p>
<p>Since the shape of <code>y</code> matches the shape of <code>x</code> then <code>BCELoss</code> won’t complain.</p>
<p>But, in the criterion the input <code>x</code> passes through the <code>model</code> which returns <code>y_hat</code> of shape <code>y_hat.shape = [batch_size, 2]</code> and from <code>sigmoid</code> we end up with <code>prob</code> which is of shape <code>prob.shape = [batch_size]</code>.</p>
<p>In other words you’re trying to pass into <code>BCELoss</code>  the vector <code>x</code> of shape <code>x.shape = [batch_size]</code> and the matrix <code>y</code> of shape <code>[batch_size, 4096]</code>. Change <code>y</code> into a vector of shape <code>y.shape = [batch_size]</code>  and everything will work again.</p>

Andrei_Cristea: <p>You’re absolutely right. I’ll fix that and try to train.</p>

Andrei_Cristea: <p>Hi, sorry for the delayed reply.</p>
<p>I don’t think there’s anything wrong with the overall method of training, but it seems like something about the architecture is the culprit. I ran an experiment where I created this network with various number of input features. You can see that when the number of input features is too high, the loss stagnates well above zero.</p>
<pre><code class="lang-auto">n_data = 40000
losses_dict = {}
for in_feat in [2, 10, 100, 1000, 4096]:
    X = sps.random(n_data, in_feat, density=0.25)
    y = (X.A[:,0] &gt; 0.5).astype(int)
    dset = SparseDataset(X, y)
    loader = tud.DataLoader(dset, batch_size=512, collate_fn=sparse_batch_collate)
    model = MLP_nodropout(in_dim=in_feat)
    losses = []
    optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
    for i, (x, y) in enumerate(loader):
        if x.is_sparse:
            x = x.to_dense()      
        y_hat = model(x).squeeze()[:,1]
        loss = nn.BCELoss(reduction='mean')(torch.sigmoid(y_hat), y)
        losses.append(loss.item())
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
    losses_dict[in_feat] = losses
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/9/691dcb031addc7aaaff0eff21788e943174cae17.png" alt="image" data-base62-sha1="eZTXRV0oQUTf0lg6zBH1NJUujS7" width="390" height="283"></p>
<p>I thought that maybe the Dropout and MaxPool layers were losing input information and leading to some “floor” loss, however I removed them and the outcome is the same.</p>

kirk86: <p>Thanks for sharing this interesting insight, it seems like for logistic regression we “kind of” have the opposite image</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/1/51065616899cd66e90ded97a51cde0ab4f8f2d55.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/51065616899cd66e90ded97a51cde0ab4f8f2d55" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/1/51065616899cd66e90ded97a51cde0ab4f8f2d55_2_345x239.jpeg" alt="image" data-base62-sha1="byMfx1w1pD8edcn7HgHAo5tOG8t" width="345" height="239" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/1/51065616899cd66e90ded97a51cde0ab4f8f2d55_2_345x239.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/5/1/51065616899cd66e90ded97a51cde0ab4f8f2d55_2_517x358.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/1/51065616899cd66e90ded97a51cde0ab4f8f2d55_2_690x478.jpeg 2x" data-dominant-color="F2F2F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">786×546 62.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I remember that when trying the same stuff with logistic regression from sckikit and solver ‘sag’ things were working.</p>
<p>Maybe this requires more investigation on what is different between scikit implementation and pytorch?</p>
<p>Also you might be right about the architecture imposing sth that makes the whole learning stagnate on large input features, otherwise this doesn’t make sense for deep learning since that’s its promise on working automatically on large feature spaces?</p>

Andrei_Cristea: <p>Another possibility would be that it will converge, but it will simply take a lot more steps to do so. Perhaps convergence would be sped up by more regularization, like maybe adding weight decay.</p>
<p>Would be curious if you discover anything, as this seems like a pretty generic model that ought to work.</p>

kirk86: <aside class="quote no-group" data-username="Andrei_Cristea" data-post="19" data-topic="153380">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/andrei_cristea/48/47843_2.png" class="avatar"> Andrei_Cristea:</div>
<blockquote>
<p>Would be curious if you discover anything, as this seems like a pretty generic model that ought to work.</p>
</blockquote>
</aside>
<p>Same question I’m asking myself as well, in the original post of showing the loss being stagnated I was using the <code>weight decay = 5e-3</code> but the only difference I can think of is that in the logistic regression from scikit learn I was using regualrisation <code>C = average(l2_norm(X[:10000], axis=1))</code>, where <code>C</code> is defined as:</p>
<pre><code class="lang-auto">C : float, default=1.0
Inverse of regularization strength; must be a positive float. Like in support vector machines, smaller values specify stronger regularization.
</code></pre>