akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Unable to install PyTorch in PyCharm using Apple Silicon

username: Omar_AlSuwaidi
cooked: <p>I’m having trouble installing PyTorch on my IDE (PyCharm). I’m not sure if I’m doing something wrong, or maybe the PyTorch package is not compiled normally through Rosetta 2, anyways I’ve been stuck on this issue for hours now! <img src="https://discuss.pytorch.org/images/emoji/apple/frowning_face.png?v=9" title=":frowning_face:" class="emoji" alt=":frowning_face:"></p>
<p>Please refer to this Stack Overflow post for more details:<br>
</p><aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/64952109/unable-to-install-pytorch-in-pycharm-python-3-9-macos" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/14492001/omar-alsuwaidi" target="_blank" rel="noopener nofollow ugc">
    <img alt="Omar AlSuwaidi" src="https://i.stack.imgur.com/VLoZ5.jpg?s=128&amp;g=1" class="thumbnail onebox-avatar" width="60" height="60">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/64952109/unable-to-install-pytorch-in-pycharm-python-3-9-macos" target="_blank" rel="noopener nofollow ugc">Unable to install PyTorch in PyCharm (Python 3.9 / macOS)</a>
</h4>

<div class="tags">
  <strong>python, python-3.x, macos, pip</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/14492001/omar-alsuwaidi" target="_blank" rel="noopener nofollow ugc">
    Omar AlSuwaidi
  </a>
  on <a href="https://stackoverflow.com/questions/64952109/unable-to-install-pytorch-in-pycharm-python-3-9-macos" target="_blank" rel="noopener nofollow ugc">08:37AM - 22 Nov 20 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Thanks for the help!</p>
Change weights of a model wrapped in amp

username: Yonathan_Aflalo
cooked: <p>Hi.</p>
<p>I have a model and an optimizer, to which I apply amp from apex package</p>
<pre><code class="lang-auto">from apex import amp
model= ...
optimizer= ...
model, optimizer = amp.initialize(model, optimizer, opt_level='O1')
</code></pre>
<p>After the model have been wrapped in amp, I would like to access one of its weights and change it. For example:</p>
<pre><code class="lang-auto">model.conv1.weight.data = new_tensor 
</code></pre>
<p>The point is when I do this, it has no effect. It looks like amp keeps a different copy of the weights, and when updating the weight on the fly, there is no effect.</p>
<p>Is there any possibility to update the weights on the flight after my model has been wrapped by amp?</p>
<p>Thanks</p>

username: Yonathan_Aflalo
cooked: <p>Anyone? I have tried to reinitilize the amp wrapper, but it is not advised by the amp documentation</p>

username: ptrblck
cooked: <p>I would not recommend to use <code>apex/amp</code> anymore, but to switch to the native implementation as described <a href="https://discuss.pytorch.org/t/torch-cuda-amp-vs-nvidia-apex/74994/3">here</a>.<br>
One reason is the added flexibility for such use cases.</p>

username: Yonathan_Aflalo
cooked: <p>Thanks a lot!! I will do this</p>
Multiplying two 3D Pytorch tensors iteratively

username: tavishjain
cooked: <p>I have two 3 dimensional Pytorch tensors, one of dimension (8, 1, 1024) and the other has dimension (8, 59, 77). I wish to multiply these two tnesors.</p>
<p>I know they cannot be multiplied in their current state, so I want to multiply them iteratively and append into a single tensor. The second tensor can be represented as (8, 59, 1) when we iterate over the 2nd dimension. In this state multiplying it with the first tensor of shape (8, 1, 1024), resulting in a tensor of shape (8, 59, 1024), and finally appending all these 77 outputs into one, resulting in the final shape of (8, 59, 1024, 77).</p>
<p>However I am having issues in it’s implementation. Can someone help me here ?</p>
When I use one cuda the momery of another is also used

username: Tomash
cooked: <p>Hello.<br>
I train my model on <code>"cuda:0"</code>, but I can see that on others GPUs my model also allocated few MBs. Do you know what could happened? Why when I use just one GPU a little bit of memory of others is also used?</p>

username: ptrblck
cooked: <p>I’m not sure if your code tries to initialize a CUDA context on all devices, but you could avoid it by masking all other GPUs via:</p>
<pre><code class="lang-auto">CUDA_VISIBLE_DEVICES=0 python script.py args
</code></pre>
How to Plot the Loss (loss values from the 'log' file) from the Training

username: num
cooked: <p>The below mentioned are the loss values generated in the file ‘log’(the iterations are actually more than this what I listed below) after train the model. Attached the screenshot of the contents of the log file for ref. How to plot the Iteration (x-axis) vs Loss (y-axis) from these contents of the ‘log’ file ?</p>
<p>0: combined_hm_loss: 0.17613089<br>
1: combined_hm_loss: 0.20243575<br>
2: combined_hm_loss: 0.07203530<br>
3: combined_hm_loss: 0.03444689<br>
4: combined_hm_loss: 0.02623464<br>
5: combined_hm_loss: 0.02061908<br>
6: combined_hm_loss: 0.01562270<br>
7: combined_hm_loss: 0.01253260<br>
8: combined_hm_loss: 0.01102418<br>
9: combined_hm_loss: 0.00958306<br>
10: combined_hm_loss: 0.00824807<br>
11: combined_hm_loss: 0.00694697<br>
12: combined_hm_loss: 0.00640630<br>
13: combined_hm_loss: 0.00593691<br>
14: combined_hm_loss: 0.00521284<br>
15: combined_hm_loss: 0.00445185<br>
16: combined_hm_loss: 0.00408901<br>
17: combined_hm_loss: 0.00377806<br>
18: combined_hm_loss: 0.00314004<br>
19: combined_hm_loss: 0.00287649</p>

username: ptrblck
cooked: <p>You could use e.g. <code>regex</code> to grab the iteration values (x-axis) and the losses (y-axis).<br>
Alternatively, e.g. <code>pandas</code> might also be a good way to load the data and get the corresponding values.</p>
Torchaudio .wav load

username: Mustapha
cooked: <p>Hello, I hope you’re all doing fine.<br>
I’m trying to preprocess .wav files with torchaudio, when i run the instruction waveform, sample_rate = torchaudio.load(filename)<br>
the waveform tensor  is of a shape [number_of_channels, some_number],<br>
sometimes the number of channels is 1 and sometimes it’s 2.<br>
I want to know whether there is a way to force the number of channels to always be one.<br>
I tried to look on the we to understand what is the meaning of the number of channel but I did’nt find anything useful.</p>

username: greaber
cooked: <p>If the two channels are identical, and even if they aren’t, you can always just use one of them.  If you have truly stereo audio, then you have to decide how you want to monoize it.  Usually just averaging the channels works OK, but it could be problematic especially if the two channels might be out of phase, which can happen due to processing or just because the microphones were far away from each other.  There are probably more sophisticated techniques than averaging that could be useful in some cases, but I never tried them.</p>

username: Mustapha
cooked: <p>That’s very helpful, thank you very much.<br>
Do you have any good reference so that I can read about waveforms?</p>
How to properly reshape a 3D tensor to a 2D for linear layer then back with new 3D's fibers corresponding to the old 3D

username: Dylan_Yung
cooked: <p>I have a 3D tensor of names that comes out of an LSTM that’s (batch size x name length x embedding size)</p>
<p>I’ve been reshaping it to a 2D to put it through a linear layer, because linear layer requires (batch size, linear dimension size) by using the following</p>
<pre><code class="lang-auto">y0 = output.contiguous().view(-1, output.size(-1))
</code></pre>
<p>this converts outputs to (batchsize * name length, number of possible characters)</p>
<p>then once I put it through the linear layer let’s call the output y0 I do this</p>
<pre><code class="lang-auto">y = y0.contiguous().view(output.size(0), -1, y0.size(-1))
</code></pre>
<p>But I’m not really sure if the cells of y are correlated properly with the cells of output and I worry this is messing up my learning, because batch size of 1 is actually generating proper names and any larger batch size is generating nonsense.</p>
<p>So what I mean exactly is<br>
outputs = (batch size * name length, embed size)<br>
y =  (batch size * name length, number of possible characters)</p>
<p>I need to make sure y[i,j,:] is the linear transformed output of outputs[i,j,:]</p>
<p>The target tensors is of (name length x correct character index) because I’m using cross entropy. So I need to ensure that every fiber of y correlates to the same index as output.</p>

username: ptrblck
cooked: <p><code>nn.Linear</code> accepts a variable number of dimensions as <code>[batch_size, *, in_features]</code>, so you might use the temporal dimension as <code>dim1</code>.<br>
The <code>view</code> operations in your code should work, since you are not permuting the dimensions (if I understand your code correctly).</p>
Runtime error after scaling

username: 111412
cooked: <h2>Hi. I was preprocessing my audio file. after scaling my mel spectogram with standard scaler , minmax (I’ve tried both) It occurs a runtime error. but the input without scaling was also float type.<br>
<img src="https://discuss.pytorch.org/images/emoji/apple/rofl.png?v=9" title=":rofl:" class="emoji only-emoji" alt=":rofl:">
</h2>
<p>RuntimeError                              Traceback (most recent call last)<br>
 in ()<br>
5 criterion_multioutput = nn.CrossEntropyLoss()<br>
6 optimizer = optim.SGD(model.parameters(), lr=0.001, momentum=0.9)<br>
----&gt; 7 model =train_model(model, criterion_multioutput, optimizer)<br>
8<br>
9 SAVE_PATH = os.path.join(os.getcwd(),‘models’)</p>
<p>11 frames<br>
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py in _conv_forward(self, input, weight)<br>
418                             _pair(0), self.dilation, self.groups)<br>
419         return F.conv2d(input, weight, self.bias, self.stride,<br>
–&gt; 420                         self.padding, self.dilation, self.groups)<br>
421<br>
422     def forward(self, input: Tensor) -&gt; Tensor:</p>
<p>RuntimeError: expected scalar type Double but found Float</p>

username: ptrblck
cooked: <p>The input tensor or the model parameters seem to be <code>DoubleTensors</code> while the others are <code>FloatTensors</code>. You could transform the data type via <code>model.float()</code> or <code>tensor.float()</code> depending which of these two objects is using the <code>double</code> type.</p>
How to quickly set multiple indicated parts of a tensor to zero

username: 111414
cooked: <p>I have a tensor A with shape of (b, 1) and a tensor B with shape of (b, h, w). The question is how could I perform this process quickly by using more efficient codes?</p>
<pre><code class="lang-auto">for i in range(b):
    B[i, int(A[i]):] = 0
</code></pre>
<p>That is, I want to set part of B (a bunch of matrices) to zero, and the beginning row indices are in A.</p>

username: ptrblck
cooked: <p>This code should work:</p>
<pre><code class="lang-python">b, h, w, = 3, 4, 4

A = torch.randint(0, h, (b, 1))
B = torch.randn(b, h, w)
C = B.clone()

idx = torch.zeros(b, h)
idx[torch.arange(b), A.squeeze()] = 1.
idx = idx.cumsum(1).bool()
B[idx] = 0.

for i in range(b):
    C[i, int(A[i]):] = 0

print((B == C).all())
&gt; tensor(True)
</code></pre>
Implementation of the very simple siamese network

username: thehemen
cooked: <p>Hello. I’m trying to implement a siamese network with a contrastive loss.<br>
It’s trained on raw tabular data. I’ve started with very a few columns to check if they’re equal (1) or not (0).<br>
The model is:</p>
<pre data-code-wrap="python3"><code class="lang-python3">class Model(nn.Module):
    def __init__(self, num_features, num_targets):
        super(Model, self).__init__()
        self.hidden_size = [5, 5, 5]
        self.dropout_value = [0.5, 0.35, 0.25]

        self.head = nn.Sequential(
            nn.BatchNorm1d(num_features),
            nn.Dropout(self.dropout_value[0]),
            nn.Linear(num_features, self.hidden_size[0]),
            nn.LeakyReLU(),

            nn.BatchNorm1d(self.hidden_size[0]),
            nn.Dropout(self.dropout_value[1]),
            nn.Linear(self.hidden_size[0], self.hidden_size[1]),
            nn.LeakyReLU(),

            nn.BatchNorm1d(self.hidden_size[1]),
            nn.Dropout(self.dropout_value[2]),
            nn.utils.weight_norm(nn.Linear(self.hidden_size[1], self.hidden_size[2]))
        )

    def forward(self, x1, x2):
        x1 = self.head(x1)
        x2 = self.head(x2)
        return x1, x2
</code></pre>
<p>And the loss is:</p>
<pre data-code-wrap="python3"><code class="lang-python3">class ContrastiveLoss(nn.Module):
    def __init__(self, margin=1.):
        super(ContrastiveLoss, self).__init__()
        self.margin = margin
        self.eps = 1e-9

    def forward(self, output1, output2, target, size_average=True):
        distances = (output2 - output1).pow(2).sum(1)
        losses = 0.5 * (target.float() * distances + \
                        (1 + -1 * target).float() * F.relu(self.margin - (distances + self.eps).sqrt()).pow(2))
        return losses.mean() if size_average else losses.sum()
</code></pre>
<p>The problem is that it’s stuck at 0,13x training and 0,2xx validation loss. Using “distances”, it gives some weird results.<br>
For the first 5 samples, instead of:</p>
<blockquote>
<p>[1. 0. 0. 1. 1.]</p>
</blockquote>
<p>It gives:</p>
<blockquote>
<p>[0. , 0. , 0.38202894, 0. , 0.]</p>
</blockquote>
<p>What could be the problem? Is this architecture appropriate? Or could it be some technical issues?</p>
How to convert syncbn to batchnormND?

username: Edwardmark
cooked: <p>I want to run a model with syncBn in cpu, so I have to convert syncBN to batchNormND, how can I do that?<br>
I just found a way to convert from bn to syncbn, but how to do the opposite? Thanks in advance.<br>
<a href="https://pytorch.org/docs/stable/generated/torch.nn.SyncBatchNorm.html?highlight=sync#torch.nn.SyncBatchNorm.convert_sync_batchnorm" rel="noopener nofollow ugc">convert2syncbn</a></p>

username: ptrblck
cooked: <p>I think the best approach would be to not use <code>convert_sync_batchnorm</code> in your model, which should be executed on the CPU.<br>
Would this work or why do you have to apply this transformation?</p>
Tansor swapping

username: alam4545
cooked: <p>How to make the following in simple way?</p>
<p>a = torch.zeros(4,3)<br>
t = torch.tensor([[1, 2, 3], [4, 5, 6], [7, 8, 9], [10, 11, 12]], dtype=torch.float)<br>
index1 = torch.tensor([1, 3])<br>
index2 = torch.tensor([0, 2])<br>
a.index_copy_(0, index1, t[index2])</p>
<p>Output:<br>
tensor([[0., 0., 0.],<br>
[1., 2., 3.],<br>
[0., 0., 0.],<br>
[7., 8., 9.]])</p>

username: ptrblck
cooked: <p>Your code looks already quite simple. I’m unsure which part you would like to remove.</p>
While or map or ... instead of for

username: Ohm
cooked: <p>Please give me an alternative way to speed up the following part of code:</p>
<pre><code class="lang-auto">    for i, data in enumerate(trainloader, 0):
        inputs, labels = data[0].to(device), data[1].to(device)
        optimizer.zero_grad()
        outputs = net(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()
        running_loss += loss.item()

        if i == 10:
           break

</code></pre>

username: ptrblck
cooked: <p>To speed up the data loading, you could preload the complete dataset assuming that you won’t apply any on-the-fly data augmentation. Since the <code>DataLoader</code> can already use multiprocessing to load and process the data in the background while the model is training, I don’t see any way to speed it up using <code>while</code> or a <code>map</code> besides trying to remove potential bottlenecks in the data loading itself.</p>
Deleting intermediate activations

username: lkp411
cooked: <p>Hey there,</p>
<p>Does anyone know if there exists a way to delete intermediate activations during forward propagation? More specifically, I only want to delete the intermediate activation data but retain the grad_fn. Assume for now, that I have a way of introducing the activations back when they are needed during back prop to compute gradients.</p>
<p>Please let me know if any part of my question doesn’t make sense.</p>
<p>Thanks!</p>
Loss and metrics both getting lower

username: hao_guo
cooked: <p>I am working on vessel segmentation.I use bcewithlogit as loss funtion and accuracy,precision,recall,specific,F1,and auc as metric.<br>
The loss keep going lower during training,but the most of these metrics goes down too.<br>
I don’t understand how could this happen,can anyone explain this,please,thank you</p>
Register_forward_hook's question

username: 6fb4b64ff0a68bcbf8fa
cooked: <p>Hello!</p>
<p>I have a question about register_forward_hook.</p>
<p>I make a RetinaNet’s model.</p>
<p>I use Resnet50’s pretrained model.</p>
<p>and hook layer2,layer3,layer4’s feature using register_forward_hook.</p>
<p>when I training this features, this features can train auto backpropagation??</p>
<p>If i don’t use register_backward_hook, this can be training??</p>
<p>thank you.</p>
<p>-James</p>

username: ptrblck
cooked: <p>I’m not sure to understand your question completely.<br>
Generally, if you store some forward activation without detaching them from the computation graph (by calling <code>.detach()</code>), you will be able to use these activations in further operations and call <code>.backward</code> on them.<br>
If you just want to inspect some intermediate activations, you should store them as their detached version.</p>
[detectron2]each time I added the RandomCrop augmentation loss function became Nan or Infinity

username: Jake_Eum
cooked: <p>here is My personal mapper for augmentation</p>
<pre><code class="lang-auto">def mapper(dataset_dict):
    dataset_dict = copy.deepcopy(dataset_dict)  # it will be modified by code below
    image = utils.read_image(dataset_dict["file_name"], format="BGR")    

    transform_list = [
                     #T.Resize((800,800)),
                     T.ResizeShortestEdge(short_edge_length=(640, 672, 704, 736, 768, 800), max_size=1333, sample_style='choice')
                     ,T.RandomApply(T.RandomFlip(prob=0.5, horizontal=True, vertical=False),prob=0.3)
                     ,T.RandomApply(T.RandomRotation((-10,10)),prob=0.4)
                     ,T.RandomApply(T.RandomSaturation(0.8, 1.2),prob=0.3)
                     ,T.RandomApply(T.RandomBrightness(0.8, 1.2),prob=0.2)
                     ,T.RandomApply(T.RandomContrast(0.6, 1.3),prob=0.2)
                     ,T.RandomApply(T.RandomLighting(0.7),prob=0.4)

                      ]
    image, transforms = T.apply_transform_gens(transform_list, image)
    dataset_dict["image"] = torch.as_tensor(image.transpose(2, 0, 1).astype("float32"))

    annos = [
        utils.transform_instance_annotations(obj, transforms, image.shape[:2])
        for obj in dataset_dict.pop("annotations")
        if obj.get("iscrowd", 0) == 0
    ]

    instances = utils.annotations_to_instances(annos, image.shape[:2])
    dataset_dict["instances"] = instances
    return dataset_dict
</code></pre>
<p>Here so far no problem<br>
but when I add the</p>
<p><code>                     #,T.RandomApply(T.RandomCrop('relative_range', (0.4, 0.6)),prob=0.05)</code><br>
to the augmentation loss become diverged and show Nan or infinity number.</p>
<p>and here is my configuration for detectron2</p>
<pre><code class="lang-auto">import os
from detectron2.data import DatasetMapper, build_detection_train_loader   # the default mapper
import detectron2.data.transforms as T
## Adding the new CascadeDropoutROIHeads to the config file

cfg = get_cfg()
cfg.merge_from_file(model_zoo.get_config_file("Misc/cascade_mask_rcnn_R_50_FPN_3x.yaml"))
cfg.DATASETS.TRAIN = ("kogas_train",)
cfg.DATASETS.TEST = ("kogas_test",)
cfg.TEST.EVAL_PERIOD = 200

cfg.DATALOADER.NUM_WORKERS = 4
cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url("Misc/cascade_mask_rcnn_R_50_FPN_3x.yaml")  # Let training initialize from model zoo
cfg.SOLVER.IMS_PER_BATCH = 12#6

cfg.SOLVER.MAX_ITER = 15000    # Need to train longer
cfg.MODEL.RESNETS.DEPTH = 50 #RESNET 50,101
cfg.SOLVER.CHECKPOINT_PERIOD = 200

#cfg.MODEL.ROI_HEADS.NAME= "CascadeROIHeads"

cfg.MODEL.ROI_HEADS.NUM_CLASSES = 4  # 4 classes
cfg.MODEL.MASK_ON = False
cfg.OUTPUT_DIR = save_path

cfg.SOLVER.LR_SCHEDULER_NAME = 'WarmupCosineLR'
cfg.SOLVER.BASE_LS = 0.0001

os.makedirs(cfg.OUTPUT_DIR, exist_ok=True)    
dataloader = build_detection_train_loader(cfg, mapper=mapper2)
trainer = CocoTrainer(cfg)
trainer.resume_or_load(resume=False)
</code></pre>
Torchscript, Using ModuleList and checking module type results in error

username: Kevin_Koehncke
cooked: <p>Hi All,</p>
<p>Trying to use Torchscript to convert my model and running into an issue, would like to know what best way forward is.</p>
<p>Say you have <code>nn.Module</code> as follows:</p>
<pre><code class="lang-auto">class Model(nn.Module):
    def __init__(self, hidden_size, input_size, num_layers, dropout):
        super(Model, self).__init__()
        self.hidden_size = hidden_size
        self.input_size = input_size
        self.num_layers = num_layers
        self.dropout = dropout

        modules = []
        for layer in range(self.layers):
            lstm = nn.LSTM(self.input_size if layer == 0 else self.hidden_size * 2, self.hidden_size, bidirectional=True, batch_first=True)
            modules.append(lstm)
            if layer != 0:
                linear = nn.Linear(self.hidden_size * 2)
                modules.append(linear)
            if layer != self.layers - 1:
                dropout = nn.Dropout(p=self.dropout)
                modules.append(dropout)
        self.modulelist = nn.ModuleList(modules)

    def forward(self, X):
        for module in self.modulelist:
          if ".LSTM" in str(type(module)):
              pack_sequence(...)
              # Do something ...
              pack_sequence(...)
          elif ".Linear" in str(type(module)):
              # Do something else ...
          elif ".Dropout" in str(type(module)):
              # Do something else ...
          return X
</code></pre>
<p>Resulting in a <code>ModuleList</code> with layer structure:</p>
<pre><code class="lang-auto">  LSTM -&gt; Dropout -&gt; LSTM -&gt; Linear -&gt; Dropout -&gt; LSTM -&gt; Linear -&gt; Dropout -&gt;  LSTM -&gt; Linear
</code></pre>
<p>I get an error when calling <code>scripted_module = torch.jit.script(Model(hidden_size=256, input_size=512, layers=4, dropout=0.3))</code> when trying to check the type of the module when iterating through the <code>ModuleList</code> during the forward pass.</p>
<p>Is there a way to circumvent this issue i.e. is this kind of forward-pass supported by TorchScript? Thanks!</p>
Grayscale/1-channel model input

username: nickvu
cooked: <p>I’m struggling to find online any information about iOS Pytorch inference with a model that expects a single channel/grayscale input tensor, as opposed to an RGB one. Does anyone have any resources or insight here?</p>

username: NAVEEN_PALURU
cooked: <p>I solved it this way: I stacked 3 copies of my input so that it has three channels and then I made the first operation as x = torch.mean(x,dim=1) in the forward method of the DL model…! The results from the app and from the computer are almost similar.</p>

username: NAVEEN_PALURU
cooked: <p>You need take care regarding the values of mean and std… accordingly…!!</p>
nn.Transformer decoder question

username: Inter
cooked: <p>If I’m trying to do sentence translation with Pytorch, what do I pass to nn.TransformerDecoder after I pass the sentence as src in nn.TransformerEncoder? Do I have to forward the output of the decoder into the input one word at a time until I get an end token?<br>
Thanks</p>

username: zhangguanheng66
cooked: <p>I’m not the expert for sentence translation. But if my understanding is correct, you could use nn.Transformer. Pass the sentence as src to encoder. The output of encoder and target should be passed to the decoder.</p>

username: shamoons
cooked: <p>Can anything be used as a start token?</p>

username: ecdrid
cooked: <p>yes, ensure it’s not part of your i/p’s.</p>
Looking for a better way to split up input data to conv1d so I don’t run out of RAM

username: Keith72
cooked: <p>I am using PyTorch to process audio data through conv1d layers, however, I am running out RAM (only 8GB available). I have an input .wav and a target .wav for the network to learn, and each file is 40MB (about 4 minutes of audio).</p>
<p>In this model, one sample of audio is learned from the previous 200 samples. (Input 200 samples and output 1 sample). In order to accomplish this, I am taking (for example) the input 8000000 samples and “unfolding” it into (8000000, 200, 1), where each audio sample becomes an array of the previous 200 samples. I then train on “unfolded_input_samples” as the training data “target_samples” as the validation data.</p>
<p>The problem is I quickly run out of RAM when unfolding the input data. Is there a way around creating this massive array while still telling PyTorch to use the previous 200 samples for each output data point? Can I break up the unfolded input array into chunks and train on each part without starting a new epoch? Or is there an easier way to accomplish this using some kind of built in method in Pytorch. Thanks!</p>

username: superunification
cooked: <p>I’m not sure of the motivation for this ‘unfolded’ tensor. The data is 99.5% redundant, and the 200-sample window can be created at train-time from the audio-data tensor. Just take a slice <code>audio[i:i+200]</code> of the 8m audio tensor, and <code>audio[i+201]</code> as your target.<br>
The only extra ingredient, then, is the set of offset indices into your audio data, which you could sample randomly or sequentially.</p>

username: Keith72
cooked: <p>That’s exactly what I’m looking for, thanks! I should have been more specific, I’m actually using PyTorch lightning, so I may need make a lower level PyTorch training loop to index the data in that way during training.</p>
<p>Thanks for the quick response!</p>
Model predicts most frequent words

username: utkuumetin
cooked: <p>Hello, i try to make bert-like model with nn.TransformerEncoder but when i predict masked word in sequence, usually its get the most frequent words in vocab.</p>
<p>When I thought about, it kind of made sense. Predicting the most frequent class for imbalanced data gives high accuracy ‘for free’.</p>
<p>What can i do about this situation, i tought changing loss function?</p>
Tcmalloc Output when training with large dataseta

username: moreshud
cooked: <p>I currently have this output <code>tcmalloc: large alloc 1073741824 bytes == 0x21728c000 @  0x7f6508e9fb6b 0x7f6508ebf379 0x7f64b9d8c74e 0x7f64b9d8e7b6 0x7f64f47f8d53 0x7f64f417354a 0x7f64f44cdc0a 0x7f64f44f5803 0x7f64f467bb14 0x7f64f47b84ee 0x7f64f4211976 0x7f64f4212b30 0x7f64f44cfb09 0x7f64f3d4e249 0x7f64f4668ae8 0x7f64f45748a5 0x7f64f421441b 0x7f64f47047d8 0x7f64f3d4e249 0x7f64f4668ae8 0x7f64f45749f5 0x7f64f5b48997 0x7f64f3d4e249 0x7f64f4668ae8 0x7f64f45749f5 0x7f650438e30e 0x50a4a5 0x50cc96 0x508cd5 0x594a01 0x59fd0e ^C</code> when training on my full dataset. The dataset is a volumetric CT scan from which I extracted its slices for segmentation on 2D UNet model. I am using Colab for the setup</p>

username: ptrblck
cooked: <p>This should be a warning not an error, which informs you about large memory allocations, which might be expected in your use case.<br>
You could change the <code>TCMALLOC_LARGE_ALLOC_REPORT_THRESHOLD</code> env variable to a higher value in case you want to remove these warnings (it should be set to 1GB by default if I’m not mistaken).</p>
Limited JIT enabling?

username: googlebot
cooked: <p>What I want to do, is run a script with PYTORCH_JIT=0, but still have some free functions optimized (i.e. opt-in mode). My quick test suggests that the following works:</p>
<pre><code class="lang-auto">torch.jit._state.enable()
@torch.jit.script
def f(x):
	return x.square()
torch.jit._state.disable() #TODO:restore
</code></pre>
<p>Is this reliable or some global objects will be messed up?</p>

username: googlebot
cooked: <p>Apparently, fuser doesn’t work this way. I’ll just resort to skipping some compilations then.</p>
"ModuleNotFoundError: No module named torchvision"

username: miner_tom
cooked: <p>I have seen several posts concerning this but the fixes did not seem to work for me.</p>
<p>I did the following:</p>
<ol>
<li>Installed python version 3.7</li>
<li>Created an environment with “conda create -n pytorchbook anaconda”</li>
<li>Conda activate pytorchbook (this works)</li>
<li>Installed several packages “conda install pytorch torchvision cudatoolkit=10.2 -c pytorch” (worked)</li>
<li>python</li>
<li>Python 3.7.9 (default, Aug 18 2020, 06:22:45)</li>
<li>
<blockquote>
<blockquote>
<blockquote>
<p>import torch (works)</p>
</blockquote>
</blockquote>
</blockquote>
</li>
<li>
<blockquote>
<blockquote>
<blockquote>
<p>import torchvision<br>
Traceback (most recent call last):<br>
File “”, line 1, in <br>
ModuleNotFoundError: No module named 'torchvision</p>
</blockquote>
</blockquote>
</blockquote>
</li>
</ol>
<p>After taking some suggestions, like loading with pip, etc, which still failed (not the install but the import), I removed the pytorchbook environment and started again from a clean slate.</p>
<p>Not sure of where to go from here.</p>
<p>Thank You<br>
Tom</p>

username: ptrblck
cooked: <p>Could you post the install log after running <code>conda install...</code>, as it should show if and which <code>torchvision</code> version was installed?</p>
How to predict output of one image in Pytorch

username: Raja_Narasimhan
cooked: <p>So me and y friends wanted to make a webapp to be hosted in heroku but heroku is not supporting latest version of pytorch in its build. So we are not able to use " model.load_state_dict(torch.load(‘alzeimer.pt’)) " in our code for predicting the output of one image. Is there any way we can do this for torch 1.3.1?? Thanks in advace</p>

username: ptrblck
cooked: <p>Are you seeing an error while trying to load the <code>state_dict</code> because it’s incompatible with the old PyTorch version? If so, could you post the error message?</p>
Google Colab RuntimeError: mat1 dim 1 must match mat2 dim 0

username: HongyuShen
cooked: <p>Hi, I am studying NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis, and I found a very useful Pytorch implementation(<a href="https://github.com/krrish94/nerf-pytorch" rel="noopener nofollow ugc">https://github.com/krrish94/nerf-pytorch</a>) on Github. I encountered a problem when running the full code version(<a href="https://colab.research.google.com/drive/1L6QExI2lw5xhJ-MLlIwpbgf7rxW7fcz3" rel="noopener nofollow ugc">https://colab.research.google.com/drive/1L6QExI2lw5xhJ-MLlIwpbgf7rxW7fcz3</a>) of NeRF on Google Colab, all the cells are working fine except for the last cell(below Run training / validation cell).</p>
<p>I got the following error at line 113:</p>
<p>RuntimeError: mat1 dim 1 must match mat2 dim 0</p>
<p>It seems like the source code is missing two arguments in nerf_forward_pass function, the nerf_forward_pass function has 19 arguments and it inputs only 17, but I still got the same error after adding the two missing arguments. How do I solve this problem?</p>
<p>I would be glad if you can provide guidelines or solutions.</p>

username: ptrblck
cooked: <p>Could you post the complete stack trace of the error message?<br>
I guess a linear layer might be using a wrong number of <code>in_features</code> for the incoming activations and will thus yield this shape mismatch.</p>

username: HongyuShen
cooked: <p>I contacted the author, he said he enabled the public editing. I think that’s why this error occurred. It is very likely that someone mistakenly edited the code. But recently the author is not available, so it would be long until he comes back to fix it.</p>

username: HongyuShen
cooked: <pre><code class="lang-auto">RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-15-885795271e1c&gt; in &lt;module&gt;()
    111         num_fine=num_fine, mode="train", lindisp=False, perturb=True,
    112         encode_position_fn=encode_position_fn,
--&gt; 113         encode_direction_fn=encode_direction_fn
    114     )
    115 

7 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in linear(input, weight, bias)
   1688     if input.dim() == 2 and bias is not None:
   1689         # fused op is marginally faster
-&gt; 1690         ret = torch.addmm(bias, input, weight.t())
   1691     else:
   1692         output = input.matmul(weight.t())

RuntimeError: mat1 dim 1 must match mat2 dim 0
</code></pre>

username: ptrblck
cooked: <p>Add print statements to show the shape of each activation before feeding it into a linear layer in the <code>forward</code> function via:</p>
<pre><code class="lang-python">x = ...
print(x.shape)
x = self.fc(x)
</code></pre>
<p>This would tell you which activation is causing this error and if a reshaping is done in a wrong way or if the <code>in_features</code> of the linear are set to a wrong value.</p>
Is it possible to load checkpoint of torch in pytorch?

username: deJQK
cooked: <p>I have a checkpoint file which was trained with torch and the file extension is t7. Is it possible for me to load it in a pytorch model? Thanks.</p>

username: albanD
cooked: <p>Hi,</p>
<p>Very old versions of pytorch used to have a compatibility layer to do this but it was removed a while ago.<br>
It might be simpler for your to dump the file in json from lua and reload it in python. It will be a bit slow but should be fairly simple.</p>

username: deJQK
cooked: <p>Thanks a lot <a class="mention" href="/u/alband">@albanD</a></p>
Set random seed for a given network

username: smu226
cooked: <p>Hello! I tried for a while to initialize my network using a fixed random seed, but it doesn’t seem to work (if I run the code multiple times and print the parameters, they are different every time). Here is what I have so far:</p>
<pre><code class="lang-auto">torch.cuda.manual_seed_all(0)
class Encoder(nn.Module):
    def __init__(self):
        super().__init__()

        N = 32
        self.encoder = nn.Sequential(
            nn.Conv2d(3, N, 4, 2, 1),
            nn.ReLU(True),
            nn.Conv2d(N, N, 4, 2, 1),
            nn.ReLU(True),
            nn.Conv2d(N, 2*N, 4, 2, 1),
            nn.ReLU(True),
            nn.Conv2d(2*N, 2*N, 4, 2, 1),
            nn.ReLU(True),
            nn.Conv2d(2*N, 8*N, 4, 1),
            nn.ReLU(True),
            View((-1, 8*N*1*1)),
            nn.Linear(8*N, 4),
        )

    def forward(self, x):
        z = self.encoder(x)
        return z

model_E = Encoder().cuda()
</code></pre>
<p>I tired to place <code>torch.cuda.manual_seed_all(0)</code> at different places in my code, but it seems like no matter where I put it, it doesn’t make a difference. How should I do it the right way? Thank you!</p>

username: Abhilash_Srivastava
cooked: <p>Add this at the starting of the first file you call (eg: main.py)</p>
<pre><code class="lang-auto">SEED = 1234

random.seed(SEED)
np.random.seed(SEED)
torch.manual_seed(SEED)
torch.backends.cudnn.deterministic = True
</code></pre>
Getting error on Google Colab : invalid index of a 0-dim tensor. Use tensor.item() to convert a 0-dim tensor to a Python number

username: Gautham_SV
cooked: <p>Hi All,<br>
Can someone please help with the below error.<br>
I have no idea what went wrong and here is my error and  code link from my Github please help in fixing the error.</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/Batmancity/Detection/blob/main/Code.py" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/Batmancity/Detection/blob/main/Code.py" target="_blank" rel="noopener nofollow ugc">Batmancity/Detection/blob/main/Code.py</a></h4>
<pre><code class="lang-py">from .imports import *
from .torch_imports import *
from .core import *
from .layer_optimizer import *

def cut_model(m, cut):
    return list(m.children())[:cut] if cut else [m]

def predict_to_bcolz(m, gen, arr, workers=4):
    arr.trim(len(arr))
    lock=threading.Lock()
    m.eval()
    for x,*_ in tqdm(gen):
        y = to_np(m(VV(x)).data)
        with lock:
            arr.append(y)
            arr.flush()

def num_features(m):
    c=children(m)
</code></pre>

  This file has been truncated. <a href="https://github.com/Batmancity/Detection/blob/main/Code.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: ptrblck
cooked: <p>As the error message suggests, you should use <code>tensor.item()</code> instead of <code>tensor[0]</code>.<br>
Also, don’t use the <code>.data</code> attribute, as it might yield unwanted side effects.</p>
About the ways to get the length of a dataser

username: Chen0729
cooked: <p>Hi , All</p>
<p>I am not sure about the best way ofgetting the length of a dataset,</p>
<p>Shall I do, dataset.size().value() or sizeof(dataset)</p>
<p>Thanks.</p>

username: Henry_Chibueze
cooked: <p>If the dataset is a numpy array or tensor then u can simply use: <code>dataset.shape</code>.<br>
It’ll return a tuple with the shapes of the dataset at respective axis/dimensions, the 1st value of the tuple is the length of the dataset.</p>
<p>To get only the length of the dataset, u can use <code>dataset.shape[0]</code>.</p>

username: ptrblck
cooked: <p>Assuming you would like to get the length of <code>torch.utils.data.Dataset</code> then <code>print(len(dataset))</code> should work:</p>
<pre><code class="lang-python">class MyDataset(Dataset):
    def __init__(self):
        self.data = torch.randn(100, 1)
        
    def __getitem__(self, index):
        x = self.data[index]
        return x
    
    def __len__(self):
        return len(self.data)

dataset = MyDataset()
print(len(dataset))
&gt; 100
</code></pre>

username: Chen0729
cooked: <p>Thanks a lot. Much appreciated.</p>
Transformer masks explanation?

username: Vadim
cooked: <p>Can somebody please point me to a tutorial with a clear explanation of what each of the TransformerEncoder/Decoder mask parameters do, and when should one use them?</p>
<p>Specifically,</p>
<ul>
<li>Which mask should I use for invalid tokens in  TransformerEncoder input?</li>
<li>Same for invalid tokens in TransformerDecoder input?</li>
<li>Which mask should I use to deal with invalid “memory” entries I need to pass to TransformerDecoder?</li>
</ul>
<p>So far I tried <code>src_key_padding_mask</code>, <code>tgt_key_padding_mask</code> and <code>memory_key_padding_mask</code> respectively, but I am getting output tensors consisting entirely of NaNs.</p>
<p>Thanks!</p>

username: Abhilash_Srivastava
cooked: <p>This is the one I usually refer to: <a href="https://pytorch.org/tutorials/beginner/transformer_tutorial.html" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/transformer_tutorial.html</a></p>
Concat images iteratively

username: mvaccaro
cooked: <p>Hi everyone, suppose I’ve two tensors holding N RGB images of size 16x32 so:</p>
<p>Input_0 = N × 3 × 32 × 16<br>
Input_1 = N × 3 × 32 × 16</p>
<p>now I want to iteratively concat these images (e.g first image of input_0 with all N images of input_1, second image of input_0 with all N images of input_1, and so on…). In order to get an output like:</p>
<p>Output = N × N × 3 × 32 × 32</p>
<p>How can I do that?</p>
Tuturial pytorch_with_examples, parameters passed to torch.optim

username: ap3rg
cooked: <p>Hi, I am going through the beginners examples, and on the part of PyTorch: optim, it is not clear to me why the optimisation package receives all model parameters (including the Tensors that describe the states of the layers) instead of only the Tensors that describe the weights.  Maybe I am missing a more fundamental concept, but it was my understanding that we just wanted to updated the weights.</p>
<pre><code class="lang-auto">optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
</code></pre>
<p>Thank you!</p>

username: ptrblck
cooked: <p><code>model.parameters()</code> will return all registered <code>nn.Parameters</code> not the arguments to layers.<br>
You can check it via:</p>
<pre><code class="lang-python">print(list(model.parameters()))
# or
print(dict(model.named_parameters()))
</code></pre>
Automatic Mixed Precision Sum of different losses

username: Maximal
cooked: <p>Hi,</p>
<p>I have a question regarding the mixed precision training when using a more complex loss that is the sum of individual loss terms. The mixed precision tutorial says that you have to call the scaler on each of the individual losses, however, the example also uses different models and optimizers.<br>
If I only have a single model and want to optimize the sum of two losses, eg some additional regularization term on top of cross-entropy, is there a difference between calling the scaler on the sum vs calling the scaler on each of the individual losses?</p>
<p>Thank’s a lot,<br>
Max</p>

username: ptrblck
cooked: <p>You could use a single <code>GradScaler</code> and scale the final loss as described <a href="https://discuss.pytorch.org/t/whats-the-correct-way-of-using-amp-with-multiple-losses/93328/3">here</a>.</p>
Train the same neural network with multiple datasets

username: Username2
cooked: <p>Hey,</p>
<p>I built a CNN-LSTM model to forecast the monthly demand of some product item in the future (1, 3 and 6 month in the future) based on the sales and order history + some chosen indicators.<br>
(It is kind of a time series except for some dates I have multiple entries…)</p>
<p>The data consists of the sales history for many, many items from different product groups (for the last 7 years). It is a mix of categorical and continuous data - I included embedding layers in the preparation of the data to take care of the categorical features.</p>
<p>So far I trained and tested the NN with subsets of the main dataset, only containing information for some selected item and some shortened time frame (3-4 years).<br>
This works quite well for the moment (just a little bit overfitting that needs to be taken care of).</p>
<p>Is there a possibility to train the same neural net on multiple different data sets, e.g. different items or different time frames and combine this “knowledge” to one model.</p>
<p>I can’t just feed the whole dataset to the model and adjust the output, because the data will  be too big.</p>
<p>To make the input the same length, I had to pad my sequences, which makes the input data even bigger. And if the data is too big my computation crashes when I want to create the dataset that I need for my dataloaders. (all available ram is used…)</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Username2" data-post="1" data-topic="103117">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/username2/48/30150_2.png" class="avatar"> Username2:</div>
<blockquote>
<p>And if the data is too big my computation crashes when I want to create the dataset that I need for my dataloaders. (all available ram is used…)</p>
</blockquote>
</aside>
<p>Would it be possible to lazily load the data, i.e. each call into <code>Dataset.__getitem__</code> would load a single sample and the <code>DataLoader</code> would create the final batch?</p>