akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to validate autoencoder model after training

username: 2hyes
cooked: <p>Hello.<br>
I want to detect the fabric defect using autoencoder.<br>
Actually, data size is too small to do ML, so I cropped images to patches.<br>
number of nodefect images: 85<br>
number of patch of a image: 172<br>
Then I trained the autoencoder.</p>
<p>During training time, loss is computed for every patches, maybe # of losses are (num_epoch * # of images * # of patches).<br>
&lt;loss_torch = np.empty((1, num_epoch, 85, 172)) &gt; with this code, I made the np place for losses, then convert to tensor.</p>
<p>During test time, if computed loss (for every patch) &gt; threshold then the image to which this patch belongs will be detected as a defect.</p>
<p>As I divided image into patches and trained Autoencoder, I called Dataloader with (# of batches = 1).<br>
Then I wonder how I can draw a graph that reduces loss as the epoch progresses.<br>
Also, I wonder if I stop training with the status that has [1, # of epochs, # of images, # of patches] shape of losses, and then do model.eval(), proceed with the model evaluation(validate).</p>
<p>Thank you for reading long sentences.</p>

username: ptrblck
cooked: <p>I’m not sure I understand the question correctly, so please feel free to correct me.</p>
<p>Assuming you are calculating the loss for each patch, then you should be able to print this loss during training directly or alternatively calculate the mean of all patches for the particular image.<br>
However, based on your data loading implementation and shuffling, it might be challenging.</p>
<p>For evaluation you could compute the loss in the same way as during training.<br>
However, for the prediction of each image, I would recommend to process all patches of the image sequentially, so that you can yield the right class after classifying all patches.</p>

username: 2hyes
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Thank you for reading long question and reply me.</p>
<p>So you mean that during training, compute loss for every patches and then do mean(all losses).<br>
After that, during validating, compute loss and make  to classify whether the image is defect or not.</p>
<p>Am I following you?</p>

username: ptrblck
cooked: <p>I’m not sure if you really need to compute the mean of the losses of all patches from the same image and I would assume that you could directly use each patch separately.</p>
<p>During validation you would have to use all patches from the image, apply your threshold, and classify the image based on the classification of all patches corresponding to this image.</p>

username: 2hyes
cooked: <p>thank you.I’ll try as you say.</p>
When load_state_dict, strict=False do not work

username: woolpeeker
cooked: <p>My pytorch version is torch==1.5.0<br>
weight file url is: <a href="https://download.pytorch.org/models/inception_v3_google-1a9a5a14.pth" rel="noopener nofollow ugc">https://download.pytorch.org/models/inception_v3_google-1a9a5a14.pth</a></p>
<pre><code class="lang-auto">import torch
import torchvision as tv
weights = torch.load('data/pretrained_weights/inception_v3_google-1a9a5a14.pth')
net = tv.models.inception_v3(num_classes=365, pretrained=False, transform_input=False)
net.load_state_dict(weights, strict=False)
</code></pre>
<p>Then it throw me a error:</p>
<blockquote>
<p>RuntimeError: Error(s) in loading state_dict for Inception3:<br>
size mismatch for AuxLogits.fc.weight: copying a param with shape torch.Size([1000, 768]) from checkpoint, the shape in current model is torch.Size([365, 768]).<br>
size mismatch for AuxLogits.fc.bias: copying a param with shape torch.Size([1000]) from checkpoint, the shape in current model is torch.Size([365]).<br>
size mismatch for fc.weight: copying a param with shape torch.Size([1000, 2048]) from checkpoint, the shape in current model is torch.Size([365, 2048]).<br>
size mismatch for fc.bias: copying a param with shape torch.Size([1000]) from checkpoint, the shape in current model is torch.Size([365]).</p>
</blockquote>

username: woolpeeker
cooked: <p>It seems the shape mismatching always throws an exceptions. Only key mismatching can be ignored.</p>
The same model works on 2080ti but can not work on 1080ti

username: JoeJoe
cooked: <p>I am training a seq2seq model for text to speech task. And I found that when training the same model on a 1080ti and a 2080ti respectively (both with the same default config from the repository, i.e., the same batch size，the same learning rate and so on).<br>
The model on 2080ti can pick up a meaningful alignment while the model on 1080ti can not。Although the loss of the model on 1080ti is descending normally.</p>
<p>I am quite confused，is the difference between GPUs resulting that problem?  I have checked the versions of pytorch to be consistent. The code also has setted a specific random seed. So where is the problem? Anybody had encountered similar problems?</p>

username: ptrblck
cooked: <p>Could you explain what “alignment” means in this context?<br>
Is your model not predicting anything useful, while the losses decrease on both GPUs?</p>

username: JoeJoe
cooked: <p>Thanks for your reply. When training a seq2seq text-to-speech model, the alignment usually means a diagonal line that aligns the outputs with the inputs based on the attention mechanism. When a neural model picks up this alignment, it can be can be used to predict. When the neural model fails, it can not work normally(not predicting anything useful).</p>
<p>And i found that the model can be trained normally on 1080ti just now. However, It seems that the 1080ti just need more iterations to pick up this aligment compared to 2080ti. For example, when training the model with same config, the model picks up an meaningful alignment about 30k iterations on the 2080ti while the model on 1080ti needs about 50k iterations to pick up this alignment.</p>

username: ptrblck
cooked: <p>How reproducible is this effect, i.e. out of 10 runs with different seeds how often is the 1080 worse by which margin compared to the 2080?<br>
Also, I assume you are using the exact same environment?</p>

username: JoeJoe
cooked: <p>Only the software enviroment are same, which means the same CUDA version, cudnn version, pytorch version, numpy version and linux version. But they are training on different devices.</p>
<p>I am sorry that i haven’t tried it for many runs. And I have only trained the model with one specific seed, perhaps 3 runs. For each run, the 2080ti seems more “smart” to learn the alignment at a early iteration than 1080ti.</p>
Is there a way to resume logging checkpoint while using Tensorboard?

username: myidispg
cooked: <p>I am training a RL Agent using the library: <a href="https://github.com/thu-ml/tianshou" rel="nofollow noopener">https://github.com/thu-ml/tianshou</a>. This library allows visualizing results using Tensorboard. But, in cases where I have to resume training from a checkpoint, I cannot resume logging from the previous point and the logging starts again losing all the previous progress. I tried keeping the tfevent files but then the writer starts overwriting on the graph.</p>
<p>Is there any way that I could start logging into the tensorboard from the checkpoint?</p>