akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
What is the optimal way to set the learning rate?

Aiman_Mutasem-bellh: <p>Hello all <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>I have a question for the experts about setting hyperparameters like learning rate and dropout for NMT project (Multi-head attention). The size of training data is big and each training session takes a long time, so it is hard to set it during experiments. Total training data as below:</p>
<ol>
<li>Synthetic data for pre-training in 6 GB.</li>
<li>First step fine-tuning data in 20 MB.</li>
<li>Second step fine-tuning monolingual data 200 MB.</li>
</ol>
<p>I have decided to use learning rates with each dataset like <code>1e-4</code>, <code>1e-3</code>, and <code>1e-1</code> but I’m not sure these are the best options.</p>
<p>Is there any statistical method to calculate the value of learning rate and dropout?</p>
<p>The multi-head attention hyperparameters are below:</p>
<pre><code class="lang-auto">D_MODEL = 256
N_LAYERS = 4
N_HEADS = 8
HIDDEN_SIZE = 512
MAX_LEN = 400
DROPOUT = 0.15
BATCH_SIZE = 64
LR = 1e-4
N_EPOCHS = 5
</code></pre>
<p>The available server is two <code>GPUs TITAN RTX - Total GPU RAM is 50 gb</code></p>
<p>Kind regards,<br>
Aiman Solyman</p>

RaLo4: <aside class="quote no-group" data-username="Aiman_Mutasem-bellh" data-post="1" data-topic="112715">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aiman_mutasem-bellh/48/14771_2.png" class="avatar"> Aiman_Mutasem-bellh:</div>
<blockquote>
<p>Is there any statistical method to calculate the value of learning rate and dropout?</p>
</blockquote>
</aside>
<p>It’s not directly calculating, but you could try out <a href="https://optuna.org/" rel="noopener nofollow ugc">optuna</a> / <a href="https://github.com/optuna/optuna" rel="noopener nofollow ugc">optuna-github</a> which is “An open source hyperparameter optimization framework to automate hyperparameter search”<br>
A simple pytorch example can be found <a href="https://github.com/optuna/optuna/blob/master/examples/pytorch/pytorch_simple.py" rel="noopener nofollow ugc">here</a></p>

googlebot: <p>hyperparam search libs are a bit overkill for two independent parameters (you only need to perform two line searches).</p>
<p>also, some hyperparams are tunable with reduced dataset and/or epoch count</p>
<p>for LR it is common to do warmup &amp; decay scheduling instead; early peak LR can be set as high as possible, and decay parameters are tuned instead (alternatively, ReduceLROnPlateau scheduler does an auto decrease)</p>

Aiman_Mutasem-bellh: <p><a class="mention" href="/u/ralo4">@RaLo4</a>  &amp; <a class="mention" href="/u/googlebot">@googlebot</a>  Thank you so much, both comments are so valuable, I’m working on them now <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
When I used SSD to train my own data set and read the train.txt file, the following problems occurred

Foolish: <pre><code class="lang-auto">Traceback (most recent call last):
  File "D:/youyong/VS/deep-learning-for-image-processing-master/pytorch_object_detection/ssd/train_ssd300.py", line 181, in &lt;module&gt;
    main(args)
  File "D:/youyong/VS/deep-learning-for-image-processing-master/pytorch_object_detection/ssd/train_ssd300.py", line 114, in main
    val_data = get_coco_api_from_dataset(val_data_loader.dataset)
  File "D:\youyong\VS\deep-learning-for-image-processing-master\pytorch_object_detection\ssd\train_utils\coco_utils.py", line 66, in get_coco_api_from_dataset
    return convert_to_coco_api(dataset)
  File "D:\youyong\VS\deep-learning-for-image-processing-master\pytorch_object_detection\ssd\train_utils\coco_utils.py", line 17, in convert_to_coco_api
    targets = ds.coco_index(img_idx)
  File "D:\youyong\VS\deep-learning-for-image-processing-master\pytorch_object_detection\ssd\my_dataset.py", line 131, in coco_index
    xml_str = fid.read()
UnicodeDecodeError: 'gbk' codec can't decode byte 0xb9 in position 105: illegal multibyte sequence

</code></pre>
<p>How to solve it？</p>

ptrblck: <p>This error seems to be a decoding issue and might not be directly related to PyTorch.<br>
Could you check the suggestions from <a href="https://github.com/rkern/line_profiler/issues/37">this issue</a> and see, if you could specify an encoding?</p>
Large CPU file size

juliusbk: <p>The CPU-only version of pytorch used be &lt; 200 MB (see e.g. <a href="https://stackoverflow.com/questions/51730880/where-do-i-get-a-cpu-only-version-of-pytorch" class="inline-onebox" rel="noopener nofollow ugc">heroku - Where do I get a CPU-only version of PyTorch? - Stack Overflow</a>). Now it is &gt; 1 GB.</p>
<p>This seems to mainly be due to the inclusion of <code>dnnl.lib</code> and <code>mkldnn.lib</code> (Windows). Are these needed for inference? It seems I can just delete them.</p>
<p>Anyone have any good ideas for further reducing the file size of the torch library?</p>
<p>Thanks!</p>

ptrblck: <p>The CPU conda binaries seems still be be &lt;200MB for the <a href="https://anaconda.org/pytorch/pytorch/files">stable release</a> as well as the <a href="https://anaconda.org/pytorch-nightly/pytorch/files">nitghtlies</a>. Could you show a log where the &gt;1GB CPU binary is installed?</p>

juliusbk: <p>The wheel file is only 184.2 MB, but unpacked (i.e. the folder in site-packages) is 936 MB. But this is perhaps to be expeced?</p>
<pre><code class="lang-auto">(venv) λ pip install torch==1.7.1+cpu -f https://download.pytorch.org/whl/torch_stable.html
Looking in links: https://download.pytorch.org/whl/torch_stable.html
Collecting torch==1.7.1+cpu
  Using cached https://download.pytorch.org/whl/cpu/torch-1.7.1%2Bcpu-cp39-cp39-win_amd64.whl (184.2 MB)
Collecting numpy
  Using cached numpy-1.20.1-cp39-cp39-win_amd64.whl (13.7 MB)
Collecting typing-extensions
  Using cached typing_extensions-3.7.4.3-py3-none-any.whl (22 kB)
Installing collected packages: numpy, typing-extensions, torch
Successfully installed numpy-1.20.1 torch-1.7.1+cpu typing-extensions-3.7.4.3
</code></pre>

ptrblck: <p>Ah OK, thanks for the explanation.<br>
It seems you’ve already narrowed it down to the CPU acceleration libs. I’m not completely sure, if <code>dnnl</code> and <code>mkldnn</code> is packed into the <code>pip</code> wheels, you you might give it a try.</p>
How to del or re-init Dataloader?

PistonY: <p>I need to validate the train model with dynamic input size.<br>
But the size I need test is not fixed.<br>
So I want to re-init the Dataloader before I test?<br>
Does it have a way to do this safely?</p>

AladdinPerzon: <p>I think you can just re-initialize the Dataset and the DataLoader and change the size in transforms (assuming working with images?), or is there any particular reason you think this is not a good idea?</p>

PistonY: <p>Yeah I think I could change transform before a new epoch start.That’s work.</p>
Pytorch distributed calling init_rpc() -> rpc.shutdown() -> init_rpc()

jeremysalwen: <p>Is is supported to re-initialize the pytorch distributed RPC framework after it has been shut down previously, from the same process?</p>
<p>e.g.</p>
<pre><code class="lang-auto">rpc.init_rpc(...)
rpc.shutdown()
rpc.init_rpc(...)
</code></pre>

Yanli_Zhao: <p>it should work. Did you try? did it work for you?</p>

Yanli_Zhao: <p>it should work with FileStore, there is still some issue for using TCPStore if you call init_rpc() twice</p>

jeremysalwen: <p>Ah, I am using TCP communication. I was getting hangs. Is there a workaround?</p>
Multiclass classification with softmax

jpj: <p>I have 5 classes. I’m using a softmax function and getting 5 probabilities in each row that add up to 1 in total.<br>
To calculate the loss do I just pick the column with the highest probability, assign it to the column number and then calculate loss. For eg: if column 1 has highest probability- that row is classified as Class 1 and let’s say the true output is Class 0. So the MSE loss is (1-0)^2 = 1. Or is there any better method?</p>

KFrank: <p>Hi jpj!</p>
<aside class="quote no-group" data-username="jpj" data-post="1" data-topic="112740" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/ed655f/48.png" class="avatar"> jpj:</div>
<blockquote>
<p>Or is there any better method?</p>
</blockquote>
</aside>
<p>Yes, you should use <a href="https://pytorch.org/docs/1.7.1/generated/torch.nn.CrossEntropyLoss.html#torch.nn.CrossEntropyLoss" rel="noopener nofollow ugc">CrossEntropyLoss</a> as your loss criterions (and not<br>
use <code>Softmax</code> because <code>CrossEntropyLoss</code> expects <em>logits</em> rather than<br>
probabilities as its predicted <code>input</code>).</p>
<p>Best.</p>
<p>K. Frank</p>
Pad a tensor with random values

iamneerav: <p>I need to pad a 3x3 tensor on all sides with some random values sampled from either the tensor itself or from a distribution. So, a 2d tensor</p>
<pre><code>1 2 3
4 5 6
7 8 9
</code></pre>
<p>after padding becomes</p>
<pre><code>4 1 2 2 3
1 1 2 3 2
1 4 5 6 4
3 7 8 9 2
2 2 1 3 3
</code></pre>
<p>or</p>
<pre><code>0.22  0.55   0.91   0.11  0.65
0.01   1       2      3   0.99
0.23   4       5      6   0.37
0.43   7       8      9   0.41
0.89  0.13   0.25   0.71  0.93
</code></pre>
<p>Is there any way to do this in pytorch?</p>

Eta_C: <pre><code class="lang-python">result = torch.randn(5, 5)
result[1:-1, 1:-1] = x
</code></pre>