akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Apply softmax to selected batch indices based on a condition

jarvico: <p>Hi,</p>
<p>I have an DNN model for regression.</p>
<p>Assuming that the output has 3 dimensions: batch_size, row, col :</p>
<p>I want to apply softmax function to the model output (to dim=1 , rows), but only under certain condition. And my condition is if the sum across dim 1 is greater than 1.</p>
<p>So softmax function will only be apllied to selected batch indices (if any). Otherwise I dont need to apply softmax.</p>
<p>Would you please help me on how to implement this?</p>
<p>Best wishes,</p>

Salman_Abbasi: <p>Hi. you can try this</p>
<p>x=torch.rand(2, 5,2)<br>
m = nn.Linear(2, 30)</p>
<p>if torch.sum(x, 1)&gt;1:<br>
m.append(nn.Softmax(dim=1))</p>
<p>model = nn.Sequential(*m)</p>
Straight line loss vs epochs diagram

ReemY: <p>Dear All,</p>
<p>I am new to Pytroch,<br>
I am modelling a neural network for binary classification. I am obtaining a straight line in loss vs epochs. Is that right?</p>
<p>Below is my code:</p>
<p>y_loss=<span class="chcklst-box fa fa-square-o fa-fw"></span><br>
for t in range(epochs):<br>
train_loss = 0<br>
for batch, (X, y) in enumerate(train_dataloader):</p>
<pre><code>    # Compute prediction and loss
    pred = model(X)
    train_loss = loss_fn(pred, y)

    # Backpropagation
    optimizer.zero_grad()# zero the gradient buffers
    train_loss.backward()
    optimizer.step()# Does the update
    
    if batch % 100 == 0:
      y_loss.append(train_loss.item() / dataset_size)
</code></pre>
<p>plt.plot(y_loss)<br>
plt.ylabel(‘y_loss’)<br>
plt.xlabel(‘epoch’)<br>
plt.title(“Learning rate %f”%(learning_rate))<br>
plt.show()</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/b/8b1cf3455573e90863743280efb17338386dfb24.png" alt="image" data-base62-sha1="jQEoqOUCc48e5AwP3WSczifb6AY" width="412" height="278"></p>

Stefano_Chioccarello: <p>In my opinion is not right. A correct plot of the losses should exhibit a descending curve, to demonstrate the fact that over the epochs the loss decreased hence the network was learning properly.  There must be something wrong with the computation of the loss, because it is very weird that is never changes over 2000 epochs.</p>
<p>What loss function are you using?</p>
<p>Moreover, note that the title of the plot is no sense: the learning rate is an hyperparameter regarding the optimizer, and it affects the step that the gradient descent takes (i.e. “acceleration” with which the algorithm run). I suggest you to explore a bit more the theory behind the different elements of a neural network training phase.</p>

ReemY: <p>Thank you for your reply.<br>
I am using binary crossentropy loss function.<br>
it is a binary classification FCNN problem.</p>

Salman_Abbasi: <p>Have you checked how many iterations it run</p>
<p>Have you tried appending without the condition if batch % 100 == 0</p>
How to run the dataloading on a separate machine?

Olivier-CR: <p>Hi,</p>
<p>In order to avoid a CPU bottleneck to consume GPU budget, a common idea is to offload dataloading to a fleet of CPU instances, and send clean batches to the fleet of GPU instances. This is for example documented here:</p>
<ul>
<li><a href="https://aws.amazon.com/blogs/machine-learning/improve-price-performance-of-your-model-training-using-amazon-sagemaker-heterogeneous-clusters/" rel="noopener nofollow ugc">SageMaker Heterogeneous Clusters</a></li>
<li><a href="https://towardsdatascience.com/overcoming-data-preprocessing-bottlenecks-with-tensorflow-data-service-nvidia-dali-and-other-d6321917f851" rel="noopener nofollow ugc">Overcoming Data Preprocessing Bottlenecks with TensorFlow Data Service, NVIDIA DALI, and Other Methods</a></li>
</ul>
<p>TF has the TF Data Service for this. Does PyTorch has a simple solution to do this? Having the dataset + dataloader running on one fleet, and having the training loop pulling clean batches on a separate fleet?</p>
<p>the SageMaker sample does it by creating a <a href="https://github.com/aws/amazon-sagemaker-examples/blob/main/training/heterogeneous-clusters/pt.grpc.sagemaker/code/train_dnn.py#L51" rel="noopener nofollow ugc">custom Dataset class</a> pulling records from a <a href="https://github.com/aws/amazon-sagemaker-examples/blob/main/training/heterogeneous-clusters/pt.grpc.sagemaker/code/train_data.py" rel="noopener nofollow ugc">grpc service</a>, but I’m wondering if there is something more built-in.</p>

ejguan: <p>Thank you sharing your use case. We do notice such trend especially in production case and we are discussing on this topic. When we design the new TorchData project to accommodate such use case. However, since this is currently tied to how your fleet was launched and how the communication between those fleets are unknown for us, we don’t have a generalized solution for now.</p>