akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Weighted input for lstm predicting next word in word2vec model (sequence generator))

username: boris.mtdv
cooked: <p>I have a sequence generator trained to generate the next word vector from a word2vec model. The problem is some of the words appear in much grreater numbers than the others, and the model picks gets stuck in a loop of predicting only those words. I’ve read that you can add a weight parameter to some loss functions in order to weight the words that appear in greater numbers so that the model doesn’t predict them as much, but my loss function of choice is CosineEmbeddingLoss, which doesn’t have a weight parameter. The input and target of the lstm is a series of word2vec vectors. How can I weigh this input appropriately so as to achieve a more balanced prediction?</p>

username: Abhilash_Srivastava
cooked: <p>A better way to handle class imbalance would be to pre-process the input data and post-process the results.<br>
eg: if there are stopwords, which you don’t necessarily want in the results, get rid of them.<br>
I’ve used this for temperature scaling, <code>softmax = e^(z/T) / sum_i e^(z_i/T)</code>, to bring about variations in the output. Not sure if you can use it with <code>CosineEmbeddingLoss</code>.</p>
Distributed Training by Pytorch

username: Junwu_Weng
cooked: <p>I am going to train my model on multi-server (N servers), each of which includes 8 GPUs. It means that I want to train my model with 8*N GPUs.</p>
<p>I have checked the code provided by a tutorial, which is a code that uses distributed training to train a model on ImageNet. ( <a href="https://github.com/pytorch/examples/tree/master/imagenet" rel="nofollow noopener">https://github.com/pytorch/examples/tree/master/imagenet</a> )</p>
<p>I found that I need to run the training code on different server seperately just as the guide introduce :</p>
<h3>Multiple nodes:</h3>
<p>Node 0:</p>
<pre><code class="lang-auto">python main.py -a resnet50 --dist-url 'tcp://IP_OF_NODE0:FREEPORT' --dist-backend 'nccl' --multiprocessing-distributed --world-size 2 --rank 0 [imagenet-folder with train and val folders]
</code></pre>
<p>Node 1:</p>
<pre><code class="lang-auto">python main.py -a resnet50 --dist-url 'tcp://IP_OF_NODE0:FREEPORT' --dist-backend 'nccl' --multiprocessing-distributed --world-size 2 --rank 1 [imagenet-folder with train and val folders]
</code></pre>
<p>I am wondering is it possible to input the command on one server and then run the codes on different servers simultaneously and automatically?</p>
<p>Thank you!</p>

username: Junwu_Weng
cooked: <p>Any help would be appreciated !</p>

username: nexgus
cooked: <p>Maybe you can try something like <a href="https://www.ansible.com/" rel="nofollow noopener">ansible</a> to deploy your application to multiple machines.</p>

username: Yaroslav_Bulatov
cooked: <p>This depends on your cluster/cloud.</p>
<p>For AWS, I have an example here: <a href="https://github.com/cybertronai/pytorch-aws/" rel="nofollow noopener">https://github.com/cybertronai/pytorch-aws/</a><br>
Basically you set your AWS credentials and then do</p>
<p><code>python mnist_distributed.py --mode=remote --nnodes=2</code></p>
Inspecting gradients w/ backwards hooks

username: Teja10
cooked: <p>I am trying to use Pytorch to inspect the values of gradients at each layer of a simple model. I am doing this using a backwards hook at each layer. My function currently prints out the same value for input and output gradients, so clearly I am misunderstanding something. In my hook, why are the values for both input and output the same. My hook function is as follows:</p>
<pre><code class="lang-auto">def grad_hook(mod, inp, out):
    print("")
    print(mod)
    print("-" * 10 + ' Incoming Gradients ' + '-' * 10)
    print("")
    print('Incoming Grad value: {}'.format(inp[0].data))
    print("")
    print('Upstream Grad value: {}'.format(out[0].data))
</code></pre>
<p>And an example output for a linear layer:</p>
<pre><code class="lang-auto">Linear(in_features=3, out_features=5, bias=True)
---------- Gradient Values ----------

Incoming Grad value: tensor([-1.3997, -2.1604,  0.8113, -1.0236,  0.3797])

Upstream Grad value: tensor([[-1.3997, -2.1604,  0.8113, -1.0236,  0.3797]])
--------------------------------------
</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p>As specified in the doc, the backward hooks on nn.Modules are not working as expected at the moment.<br>
You can use Tensor hook to get the values you want. By adding <code>register_hook()</code> to the Tensors you want to inspect either in the forward pass code or by using an <code>nn.Module</code> forward_hook to access the input/output values of a given nn.Module.</p>

username: Teja10
cooked: <p>Thank you for your reply. I saw the bug report on this Github issue (<a href="https://github.com/pytorch/pytorch/issues/16276" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/16276</a>). I’ll use register_hook on individual tensors for now.</p>
PyTorch Installation Error on Mac, Linux

username: Rafael_R
cooked: <p>Hi,</p>
<p>I am getting an error while importing torch, things used to work fine.</p>
<p>PyTorch was installed using conda:</p>
<pre><code class="lang-auto">Python 3.6.7 | packaged by conda-forge | (default, Feb 28 2019, 09:07:38) 
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt; import torch
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/home/dd/anaconda2/envs/dd/lib/python3.6/site-packages/torch/__init__.py", line 181, in &lt;module&gt;
    from .serialization import save, load
  File "/home/dd/anaconda2/envs/dd/lib/python3.6/site-packages/torch/serialization.py", line 11, in &lt;module&gt;
    import tempfile
  File "/home/dd/anaconda2/envs/dd/lib/python3.6/tempfile.py", line 45, in &lt;module&gt;
    from random import Random as _Random
  File "/home/dd/text_sum/AlexNet/video_/examples/random.py", line 5, in &lt;module&gt;
    from torch.utils import data
  File "/home/dd/anaconda2/envs/dd/lib/python3.6/site-packages/torch/utils/data/__init__.py", line 3, in &lt;module&gt;
    from .dataset import Dataset, IterableDataset, TensorDataset, ConcatDataset, ChainDataset, Subset, random_split  # noqa: F401
  File "/home/dd/anaconda2/envs/dd/lib/python3.6/site-packages/torch/utils/data/dataset.py", line 5, in &lt;module&gt;
    from torch import randperm
ImportError: cannot import name 'randperm'
</code></pre>
<p>How to fix this issue?</p>

username: albanD
cooked: <p>Hi,</p>
<p>You might have multiple pytorch installs coexisting and causing issues.<br>
I would make sure to uninstall all pytorch installs (via conda / pip etc) in your environment and then reinstall a brand new one.</p>

username: Rafael_R
cooked: <p>I installed a brand new one on both mac &amp; linux, in a new conda environment, still get the same error <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>

username: albanD
cooked: <p>Which version do you install?</p>

username: Rafael_R
cooked: <p>cuda 10 conda install linux pytorch 1.2</p>
<p>no cuda conda install mac pytorch 1.2</p>

username: albanD
cooked: <p>Actually, looking at the stack trace I think you have a problem with local files:<br>
the python <code>tempfile</code> library, when loading picks up your local <code>random.py</code> file as python’s original <code>random library</code>.<br>
Can you import torch from another folder? If so, you will have to rename your <code>random.py</code> file I’m afraid.</p>

username: Rafael_R
cooked: <p>thanks!! wow that really was the issue. everything is fine now <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Build CUDAExtension on non-CUDA environment

username: sublee
cooked: <p><code>CUDAExtension</code> in <code>setup.py</code> may fail with an environment where CUDA isn’t installed. What is the best practice to make <code>CUDAExtension</code> compatible with a non-CUDA environment?</p>
<pre><code class="lang-python">Traceback (most recent call last):
  File "setup.py", line 23, in &lt;module&gt;
    'my_extension.cu',
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 476, in CUDAExtension
    library_dirs += library_paths(cuda=True)
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 555, in library_paths
    if (not os.path.exists(_join_cuda_home(lib_dir)) and
  File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1146, in _join_cuda_home
    raise EnvironmentError('CUDA_HOME environment variable is not set. '
OSError: CUDA_HOME environment variable is not set. Please set it to your CUDA install root.
</code></pre>

username: TomB
cooked: <p>You can’t build a CUDA extension without the CUDA headers and libraries. I think your main options would be to either require the CUDA toolkit (I don’t think you should actually need a GPU, just the toolkit), or have separate CUDA and non-CUDA versions of your extension.<br>
If going with separate versions you should be able to either have a shared codebase with separate CUDA specific stuff. Or use standard C++ tricks like defines to use the same codebase while excluding CUDA stuff on CPU only builds.<br>
From the python setup.py end you’d want a <a href="https://stackoverflow.com/questions/677577/distutils-how-to-pass-a-user-defined-parameter-to-setup-py" rel="nofollow noopener">custom setuptools command</a> to allow custom command line options to control whether you install a <code>CUDAExtension</code> or a <code>CppExtension</code>. Or just have separate python packages.</p>

username: sublee
cooked: <p>Thank you for the great answer!</p>

username: TomB
cooked: <p>No problem. Actually, while I think the basic idea of selective compilation is correct, you can do the setup side more easily by just checking for the CUDA toolkit and if detected install a <code>CUDAExtension</code> otherwise install a <code>CppExtension</code>. See torchvision for an example of how to do this nicely. Especially around <a href="https://github.com/pytorch/vision/blob/49b01e3a7d9d248e2218c2c57d55346743bfb89d/setup.py#L103" rel="nofollow noopener">here</a> where, removing all the actual config, you have this sort of thing:</p>
<pre><code class="lang-auto">def get_extensions():
    extension=CppExtension
    if (torch.cuda.is_available() and CUDA_HOME is not None) or os.getenv('FORCE_CUDA', '0') == '1':
        extension = CUDAExtension
        define_macros += [('WITH_CUDA', None)]
    return extension(define_macros)

setup(
    ext_modules=get_extensions()
)
</code></pre>
<p>Where that define is used to add in CUDA stuff in the C++. Much cleaner than my suggestion of a command.</p>
How to plot a chart that shows the number of occurrences of each values in a pytorch tensor

username: oboul
cooked: <p>I want to plot a statistical graph that shows the number of occurrences of each values in a torch tensor ?<br>
Thank you</p>

username: alex.veuthey
cooked: <p>Have a look at the histogram (<code>histc</code>) function <a href="https://pytorch.org/docs/master/torch.html?highlight=hist#torch.histc" rel="nofollow noopener">here</a>.</p>
Beginner : the difference of split data in keras and pytorch

username: astri
cooked: <p>I am a beginner in Deep Learning and try sample project in keras and pytorch. I am noticed a little difference in both framework when split data.<br>
Keras : train and test<br>
pytorch : train, test, and valid</p>
<p>could anybody explain why there is a valid data in pytorch meanwhile there is only test data in keras?</p>
<p>Thank you</p>

username: JamesTrick
cooked: <p>Hi! When using Keras and calling <code>.fit()</code> Keras there’s variable called <code>validation_split</code> which splits your training data to have a validation set. The default is 0.0, meaning the data won’t be split and you won’t have a validation set (unless you specify <code>validation_data</code>) - not a good idea.</p>
<p>Within PyTorch, there’s no fit style function so you need to write your own. I sense this is what is giving you the impression that there is no val in Keras as the train/test/val is a bit more clear in PyTorch’s case.</p>
<p>Does that help? It’s in Keras, just a bit more hidden.</p>

username: astri
cooked: <p>Thank you for your explanation. It helps a lot</p>
FFN versus RNN models

username: ironv
cooked: <p>I am building models for chronic disease prediction (binary classification) using a sequence of diagnoses, procedure and medication codes collected over a two-year period, 8 years before the confirmation of the disease.</p>
<p>Ex: training: 50k observations, 1800 codes/features<br>
sequence length ptiles  25th = 12 codes, 50th = 30 codes, 75th = 71 codes</p>
<p><strong>Model 1: FFN</strong></p>
<ul>
<li>computed matrix where columns are features and values are number of occurrence of  a code.  Built vanilla FFN.</li>
<li>ran extensive grid search. best arch has 4 hidden layers with 127 neurons in each, ReLu act<br>
and dropout of .6 between all of them.    <span class="hashtag">#param:</span> 279,785,  f1: 27.6%  train time/epoch 2.67s on V100/32GB</li>
</ul>
<p><strong>Model 2: RNN</strong></p>
<ul>
<li>sorted obs (and labels) based on sequence lengths from smallest to largest</li>
<li>(pack_sequence) =&gt; embed =&gt; pack_padded_sequence (bat_f=T) =&gt; LSTM =&gt; FFN</li>
<li>grid search.  best arch: embed dim: 50, lstm_hdim: 255, fc layer widths: 63, 31, dropout .6,  <span class="hashtag">#param:</span> 422,187, f1: 26.2%,  train time/epoch  55s  on same GPU.</li>
</ul>
<p><strong>Questions</strong></p>
<ul>
<li>
<p>the FFN has ~half the number of variables to be trained. Yet it is &gt;20x faster.  I realize that may mean nothing in light of algorithmic complexities.  But, does that make sense?  In both cases the entire instance is on the GPU (CPU is not used for anything).  For the FFN I am using the TensorDataset class.  For LSTM, it is DataSet with collate_fn for DataLoader suitably modified.  Any ideas for speeding up the training?</p>
</li>
<li>
<p>there wasn’t an expectation that the <em>sequence</em> of codes 8 years out would be important as much as the code itself. But I did think the results would be closer.  I tried with 2 LSTMs, bidirectional-LSTM, etc. etc.  Is there any other arch change worth looking at?  Attention?</p>
</li>
</ul>