akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
VOCDetection year='2007' Problem

Mahesh_Srinivas: <p>I am facing a problem when I use:<br>
torchvision.datasets.VOCDetection(’/content’, year=‘2007’, image_set=‘train’, download=True)<br>
The specific problem is that download and extraction is happening for VOCtest_06-Nov-2007.tar instead of VOCtrainval_06-Nov-2007.tar. My guess is that this is due to recent changes made to voc.py. Not an expert but I think problem might be here: <a href="https://github.com/pytorch/vision/blob/c808d163f6c65ca851db59e9966807a9220fc493/torchvision/datasets/voc.py#L88" class="inline-onebox" rel="noopener nofollow ugc">vision/voc.py at c808d163f6c65ca851db59e9966807a9220fc493 · pytorch/vision · GitHub</a> [lines 88-94]<br>
I am using the following hack for now and it seems to work:</p>
<p>torchvision.datasets.voc.DATASET_YEAR_DICT[‘2007-trainval’] = torchvision.datasets.voc.DATASET_YEAR_DICT[‘2007’]</p>
<p>dummy = torchvision.datasets.VOCDetection(’/content’, year=‘2007-trainval’, image_set=‘train’, download=True)</p>
<p>This basically downloads and extracts the right tar file and subsequent commands work.</p>

patrickwilliams3: <p>Yep good call. Seems that whenever the year is “2007” they’ve created a bug where they change the key to “2007-test” so you only ever load the test data. Might be a good idea to create an Issue for this on the PyTorch Github. If you don’t want to I would be happy to.</p>
<pre><code class="lang-auto">self.year = year

        valid_image_sets = ["train", "trainval", "val"]
        if year == "2007":
            valid_image_sets.append("test")
            key = "2007-test"
        else:
            key = year
        self.image_set = verify_str_arg(image_set, "image_set", valid_image_sets)
        dataset_year_dict = DATASET_YEAR_DICT[key]
</code></pre>
<p>And in the DATASET_YEAR_DICT we have</p>
<pre><code class="lang-auto">    '2007': {
        'url': 'http://host.robots.ox.ac.uk/pascal/VOC/voc2007/VOCtrainval_06-Nov-2007.tar',
        'filename': 'VOCtrainval_06-Nov-2007.tar',
        'md5': 'c52e279531787c972589f7e41ab4ae64',
        'base_dir': os.path.join('VOCdevkit', 'VOC2007')
    },
    '2007-test': {
        'url': 'http://host.robots.ox.ac.uk/pascal/VOC/voc2007/VOCtest_06-Nov-2007.tar',
        'filename': 'VOCtest_06-Nov-2007.tar',
        'md5': 'b6e924de25625d8de591ea690078ad9f',
        'base_dir': os.path.join('VOCdevkit', 'VOC2007')
    }
</code></pre>

Mahesh_Srinivas: <p>Thanks a lot Patrick for the affirmation. As per your feedback, I took a stab at creating an Issue here: <a href="https://github.com/pytorch/pytorch/issues/53971" class="inline-onebox" rel="noopener nofollow ugc">VOCDetection year=‘2007’ Problem · Issue #53971 · pytorch/pytorch · GitHub</a> . Please let me know if I haven’t accurately represented the problem.</p>
Transformers with scheduled sampling implementation

marcpaga: <p>I am trying to implement a seq2seq model using transformers with scheduled sampling.<br>
The method that I am trying to implement is as described <a href="https://www.aclweb.org/anthology/P19-2049.pdf" rel="noopener nofollow ugc">Mihaylova 2019</a>.</p>
<p>My question revolves around how to implement the two decoder.</p>
<p>I am doubting which of these two approaches would be the correct one (maybe none is):</p>
<ul>
<li>Forward the sample twice through the decoder, first with the gold targets and second with a mix of gold and predictions.</li>
<li>Have two separate decoders, one for the first pass and one for the second, that share weights and biases.</li>
</ul>
<p>Here is some pseudocode to illustrate each case.</p>
<p>First approach:</p>
<pre><code class="lang-auto">class TransformersScheduled(nn.Module):
    def __init__(self, encoder, decoder):
        self.encoder = encoder
        self.decoder = decoder

    def forward(self, src, trg):
        src = self.encoder(src)
        prediction = self.decoder(src, trg)
        new_trg = self.mix_gold_pred(trg, prediction, 0.5)
        prediction = self.decoder(src, new_trg)
        return prediction

    def mix_gold_pred(trg, pred, factor):
        # mix gold standard labels with predicted labels
        return new_trg

## loss is calculated on the second prediction and backpropagation is applied

</code></pre>
<p>Secod approach:</p>
<pre><code class="lang-auto">class TransformersScheduled(nn.Module):
    def __init__(self, encoder, decoder1, decoder2):
        self.encoder = encoder
        self.decoder1 = decoder1
        self.decoder2 = decoder2
    def forward(self, src, trg):
        src = self.encoder(src)
        # copy the weights from decoder2 to decoder1
        self.decoder1.weights = self.decoder2.weights
        prediction = self.decoder1(src, trg)
        new_trg = self.mix_gold_pred(trg, prediction, 0.5)
        prediction = self.decoder2(src, new_trg)
        return prediction

    def mix_gold_pred(trg, pred, factor):
        # mix gold standard labels with predicted labels
        return new_trg

## loss is calculated on the second prediction and backpropagation is applied

</code></pre>
<p>In the first case, would there be a “double” backpropagation through the decoder since it is used twice during the forward?<br>
In the second case, would there be backpropagation through the first decoder?</p>
<p>Thanks in advance for your time and help!</p>
Dataloader index keeps changing

sugh7020: <p>I have dataset class as:</p>
<pre><code class="lang-auto">class SomeDataset(Dataset):
    def __init__(self, usage='val', dataset_pickle_file='./some.pkl', skip_every_n_image=1):
        super(SomeDataset, self).__init__()

        self.to_tensor = transforms.ToTensor()

        with open(dataset_pickle_file, 'rb') as file:
            self.data_info = pickle.load(file)[usage]
        self.idx = [i for i in range(0, len(self.data_info[0]), skip_every_n_image)]
        self.data_len = len(self.idx)

    def __len__(self):
        return self.data_len

    def __getitem__(self, index):  #1
        instanceNum = self.idx[index]
        color_img = self.data_info[0][instanceNum]
        color_img = Image.open(color_img)
        color_tensor = self.to_tensor(color_img)
        d_img = self.data_info[1][instanceNum] #2
        d_img = Image.open(d_img)
        d_tensor = self.to_tensor(d_img)
        output = {'image': color_tensor, 'dt': d_tensor}
        return output
</code></pre>
<p>I am running the code in debug mode on PyCharm. The initial value of index seems to be 11936 on entering <strong>getitem</strong>(at 1).<br>
But on reaching d_img the value of index and instanceNum seems to be 1517(at 2).<br>
Isn’t the value of index supposed to be constant during a particular call to <strong>getitem</strong>? The pickle file stores urls to consecutive items of color_img and d_img, ie a particular index of d_img represents the depth of color_img with the same index. Thus when the value of index/instanceNum changes within the same call to the function I guess the result from output is combining an image and a depth image of two different instances.</p>
<p>Can you all please suggest what I might be doing wrong? Thanks!</p>

Valerio_Biscione: <p>I doubt the index is changing within the same function. Are you actually printing it?<br>
I suspect it may be due to the multiprocessing.<br>
Try to set the <code>num_workers</code> to 0 in the Data Loader and check if the index is changing now.</p>
KLDivLoss and transforming back to probabilities

jakob1427: <p>Having recently adapted targets for my model to follow a distribution pattern I am using <code>KLDivLoss</code> as my loss function. I know that for stability the PyTorch <code>KLDivLoss</code> implementation uses log-probabilities as input and hence the output of my final layer is transformed using <code>log_softmax</code>.</p>
<p>My 2 questions are:</p>
<ol>
<li>
<p>Am I correct in assuming that to convert the output of <code>log_softmax</code> back to normal softmax (i.e. a vector of probabilities representing a probability distribution) I would take the natural exponent of the output? Hence if the model output is <code>output = log_softmax(dense, dim=1)</code> then I could do <code>np.exp(output)</code> to get the vector back to the range of my y_targets??</p>
</li>
<li>
<p>Upon doing the above and training my model the output is 5 orders of magnitude smaller than the y_targets! I know the model as a whole usually works (I was using more discrete y_targets before with this model and MSE loss which worked very well). The abbreviated code is below:</p>
</li>
</ol>
<p><strong>Model Class:</strong></p>
<pre><code class="lang-auto">def build_model(self):
  '''
  Main model build code...
  '''
  self.dense = nn.Linear(self.c['hidden_size'], self.c['y_distribution_size'])
  self.loss = nn.KLDivLoss()
  self.optimizer = optim.AdamW(self.parameters(), lr=self.c['optim_lr'])

def forward(self):
  '''
  Main model forward code...
  '''
  dense = self.dense(self.dropout(self.sigmoid(x_out)))
  return log_softmax(dense, dim=1)

def fit(self):
  '''
  Main model fit code...
  '''
  self.optimizer.zero_grad()
  model_out = self(x)
  loss = self.loss(model_out, y)
  loss.backward()
  self.optimizer.step()
</code></pre>
<p><strong>Y_target Tensor</strong><br>
Shape: (N, 5)<br>
Where each N has a vector such as <code>[0.2, 0.4, 0.4, 0, 0]</code> (sum of probabilities = 1)</p>
<p><strong>Mode Output Tensor</strong><br>
<code>y_pred = model(x)</code> where y_pred example is:<br>
<code>[-12.440563, -12.761744, -13.408863, -12.697731, -12.389305]</code><br>
and I am converting it to <code>y_p_pred = np.exp(y_pred)</code> which gives an example result of: <code>[3.9548686e-06, 2.8684360e-06, 1.5017746e-06, 3.0580563e-06, 4.1628732e-06]</code><br>
(sum of probabilities = 1.554601e-05 for that particular example)!</p>
<p>So I am trying to get the model to fit the resultant <code>y_p_pred</code> to <code>y_target</code>.</p>
About converting from pytorch to C++ in looping over dataset

Chen0729: <p>Hi, All</p>
<p>I have an inquiry on converting from pytorch to C++ in looping over dataset. The python code is;<br>
unlabel_loader1 = DataLoader(self.unlabeled, batch_size = self.args.batch_size, shuffle=True, drop_last=True, num_workers = 4)<br>
for (unlabel1, _label1) in unlabel_loader1:<br>
batch_num += 1</p>
<p>I convert to C++ with libtorch as:<br>
auto unlabel1 = torch::data::datasets::MNIST("/path/MNISTv2",<br>
torch::data::datasets::MNIST::Mode::kTrain).map(<br>
torch::data::transforms::Normalize&lt;&gt;(0.13707, 0.3081)).map(torch::data::transforms::Stack&lt;&gt;());<br>
auto unlabel_loader1 = torch::data::make_data_loader<a>torch::data::samplers::SequentialSampler</a>(unlabel1, torch::data::DataLoaderOptions().batch_size(args.batch_size).workers(4).drop_last(<br>
true));</p>
<p>But I am not sure how shall I translate " for (unlabel1, _label1) in unlabel_loader1:" into C++.<br>
Any comments are appreciated.</p>
<p>Thanks in advance.</p>

ptrblck: <p><a href="https://pytorch.org/cppdocs/frontend.html#end-to-end-example">This tutorial</a> gives you an end-to-end example of the C++ Frontend using the MNIST dataset.</p>

Chen0729: <p>Thanks for the comments.</p>
<p>The pytorch code is:<br>
" unlabel_loader1 = DataLoader(self.unlabeled, batch_size = self.args.batch_size, shuffle=True, drop_last=True, num_workers = 4)<br>
unlabel_loader2 = DataLoader(self.unlabeled, batch_size = self.args.batch_size, shuffle=True, drop_last=True, num_workers = 4).<strong>iter</strong>()<br>
label_loader = DataLoader(tile_labeled, batch_size = self.args.batch_size, shuffle=True, drop_last=True, num_workers = 4).<strong>iter</strong>()</p>
<pre><code>        batch_num = 0
        for (unlabel1, _label1) in unlabel_loader1:
            batch_num += 1
            unlabel2, _label2 = unlabel_loader2.next()
            x, y = label_loader.next() "
</code></pre>
<p>I write it this way in C++:</p>
<p>int batch_num = 0;<br>
auto udata_loader = unlabel_loader1-&gt;begin();<br>
auto udata_loader2= unlabel_loader2-&gt;begin();<br>
auto ldata_loader = label_loader-&gt;begin();<br>
for (int idx = 0; idx &lt; args.iter; idx++)<br>
{<br>
std::cout &lt;&lt; “process in the iter loop” &lt;&lt; std::endl;<br>
batch_num = batch_num + 1;</p>
<pre><code>            if (udata_loader == unlabel_loader1-&gt;end())
                udata_loader = unlabel_loader1-&gt;begin();
            torch::Tensor unlabel1 = udata_loader-&gt;data;

            ++udata_loader;
            if (udata_loader2 == unlabel_loader2-&gt;end())
                udata_loader2 = unlabel_loader2-&gt;begin();
            torch::Tensor unlabel2 = udata_loader2-&gt;data;

            ++udata_loader2;
            if (ldata_loader == label_loader-&gt;end())
                ldata_loader = label_loader-&gt;begin();
            torch::Tensor x = ldata_loader-&gt;data;
            torch::Tensor y = ldata_loader-&gt;target;

            ++ldata_loader;
</code></pre>
<p>May I ask if this translation is accurate please ? Thanks.</p>
torchvision.transforms.Resize() memory crash

alphanumericnosnense: <p>Running the code below devours all available RAM:</p>
<pre><code class="lang-auto">import numpy as np
from torchvision import transforms
transform = transforms.Compose([transforms.ToTensor(), transforms.Resize(256)])
X = np.full((1,256,256),0)
Y = transform(X)
</code></pre>
<p>I was making custom masks for a segmentation problem and wanted constant tensors with various values for reasons.  I don’t <em>need</em> to resize the thing, but I wondered why this was blowing up.  According to <a href="https://pytorch.org/vision/stable/transforms.html?highlight=torchvision%20transforms%20resize#torchvision.transforms.Resize" rel="noopener nofollow ugc">the docs</a>, I don’t see a problem turning an array into a tensor and then resizing.</p>

ptrblck: <p>Images stored as numpy arrays are expected in the channels-last memory layout.<br>
For a grayscale image you could thus pass <code>X</code> as <code>[256, 256]</code> or <code>[256, 256, 1]</code>, which will return the expected output.<br>
In the current format, <code>torchvision</code> assumes that your image has a height of 1, a width of 256, and 256 channels.<br>
The <code>ToTensor()</code> operation will return a tensor of <code>[256, 1, 256]</code>, which will then be passed to <code>Resize</code>.<br>
Again, <code>Resize</code> assumes the wrong layout and will try to increase the “small side” (which is the height of 1) to 256 and scale the long side with the same factor resulting in a tensor size of:</p>
<pre><code class="lang-python">  channels * height * width * 4
= 256 * 256 * (256*256) * 4 = 17179869184 bytes = 16GB
</code></pre>
<p>which fits perfectly the error message:</p>
<pre><code class="lang-python">DefaultCPUAllocator: not enough memory: you tried to allocate 17179869184 bytes. Buy new RAM!
</code></pre>

alphanumericnosnense: <p>Thanks, channel first/second/last issues have definitely fooled me more than once.</p>
Dataset dataloader

sugh7020: <p>I have dataset class as:</p>
<pre><code class="lang-auto">class SomeDataset(Dataset):
    def __init__(self, usage='val', dataset_pickle_file='./some.pkl', skip_every_n_image=1):
        super(SomeDataset, self).__init__()

        self.to_tensor = transforms.ToTensor()

        with open(dataset_pickle_file, 'rb') as file:
            self.data_info = pickle.load(file)[usage]
        self.idx = [i for i in range(0, len(self.data_info[0]), skip_every_n_image)]
        self.data_len = len(self.idx)

    def __getitem__(self, index):
         ### TODO
    def __len__(self):
        return self.data_len
</code></pre>
<p>I am confused which of the below would be correct for the <strong>getitem</strong> function:</p>
<pre><code class="lang-auto">    def __getitem__(self, index):
        color_img = self.data_info[0][self.idx[index]]
        color_img = Image.open(color_img)
        color_tensor = self.to_tensor(color_img)
        output = {'image': color_tensor}
        return output
</code></pre>
<p>Or</p>
<pre><code class="lang-auto">    def __getitem__(self, index):
        color_img = self.data_info[0][index]
        color_img = Image.open(color_img)
        color_tensor = self.to_tensor(color_img)
        output = {'image': color_tensor}
        return output
</code></pre>
<p>Can you all also please also mention the logic behind it? Thanks!</p>

rahulvigneswaran: <p>I guess the <code>[self.idx[index]]</code> would be the correct way to go.</p>
<p><strong>The logic</strong> being that if you use the <code>self.data_info[0][index]</code> directly, then the <code>skip_every_n_image</code> would become useless. But when you use self.data_info[0][self.idx[index]], you are making use of the <code>self.idx</code> which has the data after skipping every n images.</p>
Why these module "super themselves"?

ruka: <p>I’m reading mobilenetv2 implementation <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/mobilenetv2.py" class="inline-onebox" rel="noopener nofollow ugc">vision/mobilenetv2.py at master · pytorch/vision · GitHub</a><br>
I noticed that when doing init(), some module are using super(XXX, self).<strong>init</strong>().<br>
For example, class InvertedResidual() are using<br>
super(InvertedResidual, self).<strong>init</strong>()<br>
This class inherited from nn.Module, my understanding is: it just needs to use super().<strong>init</strong>() to get its parent class(nn.Module) to do initialization, but why passing itself to super()?</p>

crcrpar: <p>You’re right, <code>super().__init__()</code> suffices in that usage in Python3. Passing itself to <code>super</code> is basically Python2 style: <a href="https://docs.python.org/2.7/library/functions.html?highlight=super#super" class="inline-onebox" rel="noopener nofollow ugc">2. Built-in Functions — Python 2.7.18 documentation</a>.<br>
Note that this is not technically correct in some details.</p>

ruka: <p>Thank you for the explanation <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"></p>
Is that an ideal way to clone the best method and fine tune?

vauthority: <p>Hi Pytorch forum! Recently I came up with an idea for my side project which is using depth estimation to build a mini car toys that can keep itself away from obstacles but I don’t know what’s a proper approach to solve real world problem in this field? Is that an ideal way to just clone the best method out there and fine tune to fit this problem or read through paper proposed method and write and train model from scratch?</p>

alexandrepires: <p>Hey there,</p>
<p>Well, it depends on what you want to focus on first, also, how much experience do you have in reading the papers and creating neural networks. If your focus is to make the mini car work and perform obstacle avoidance, then, it´s a lot better for you to clone a project and fine tune it to your needs, since it will provide to you a lot quicker proof of concept than starting from scratch. I would only suggest to read proposed methods and train it from scratch only when you are very experienced in this and know what you want to try to accomplish.</p>
<p>Also, another point why cloning and fine tune it first is better, is due to the fact that when you further develop a project, you discover new needs or deprecate others. Software is always evolving and changing. For instance, you may implement from scratch a proposed method you found, but, later on, you discover that maybe the network is not precise enough or its too big to fit your processing needs! Then what happens there? You learned how to do it, sure, but you took a not so much time efficient path(time is also a precious resource <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> )</p>
<p>Fast.ai touches the subject of how to learn new subjects, if you are interested. If you are new / semi - new, its better to fine tune networks that are already implemented and tested and, later, touching the “lower level” when you need something more.</p>
<p>A summary: 99% of the times is better to grab a working network and fine tune it. It will provide you a quicker proof of concept, you will learn many requirements on the way and change things as needed, more rapidly and time efficient.</p>
Building C++ library working on all LibTorch versions

BarclayII: <p>Hi,</p>
<p>I was wondering if it is possible to build a C++ library that works on all of LibTorch 1.5, 1.6 and 1.7, that is, link against one LibTorch version and make it work on multiple LibTorch versions.</p>
<p>I tried building one shared library with a simple function, linking it with LibTorch 1.5 and 1.6.   When I <code>dlopen</code> it after importing PyTorch 1.7 it crashes when calling <code>torch::empty</code>:</p>
<pre><code class="lang-auto">undefined symbol: _ZN3c104impl23ExcludeDispatchKeyGuardC1ENS_11DispatchKeyE
</code></pre>
<p>Linking with LibTorch 1.7 works fine.</p>
<p>My C++ library code looks like the following:</p>
<pre><code class="lang-auto">static at::Device get_device(DLContext ctx) {
  switch (ctx.device_type) {
   case kDLCPU:
    return at::Device(torch::kCPU);
    break;
   case kDLGPU:
    return at::Device(torch::kCUDA, ctx.device_id);
    break;
   default:
    // fallback to CPU
    return at::Device(torch::kCPU);
    break;
  }
}

extern "C" {

DLManagedTensor* TAempty(
    std::vector&lt;int64_t&gt; shape,
    DLDataType dtype,
    DLContext ctx) {
  auto options = torch::TensorOptions()
    .layout(torch::kStrided)
    .device(get_device(ctx))
    .dtype(at::toScalarType(dtype));
  torch::Tensor tensor = torch::empty(shape, options);
  return at::toDLPack(tensor);
}

};
</code></pre>

glaringlee: <p><a class="mention" href="/u/barclayii">@BarclayII</a><br>
To my knowledge, this is not possible at the moment. Symbols in different version of libtorch are not exactly the same. Symbol might moved, their visibility might changed, etc.</p>

BarclayII: <p>Thanks!  Then I guess I should build one library per LibTorch version.</p>

glaringlee: <p><a class="mention" href="/u/barclayii">@BarclayII</a><br>
Just wondering, what are you trying to do? Benchmarking? Will the latest libtorch fit all your needs?</p>

BarclayII: <p>The reply is extremely late but I’m trying to reuse the PyTorch CUDA allocator in my C++ library.  I intend to make my library support a list of different PyTorch versions.</p>
Cryptographically Secure RNG in Opacus

James_M: <p>Is there any literature/documentation on the necessity of cryptographically-secure RNG in Opacus? As far as I can tell it’s only used for noise generation so would like to understand the underlying risks/compromises. It looks to me like TF-Privacy isn’t using a CSPRNG for its noise generation so am curious about the reasoning for the divergence.</p>

ilyamironov: <p>Excellent question! There are two separate uses of CSPRNG in Opacus - for batch composition (shuffling or sampling) and for noise generation. If batches are formed using an insecure RNG, the algorithm is still differentially private, but we don’t get to claim amplification by subsampling.</p>
<blockquote>
<p>It looks to me like TF-Privacy isn’t using a CSPRNG for its noise generation so am curious about the reasoning for the divergence.</p>
</blockquote>
<p>It’s a pragmatic choice. Basically, TF does not natively support CSPRNG; Pytorch’s decision to implement it is in no small part due to its commitment to supporting Opacus.</p>
<p>I’ll add parenthetically, that, strictly speaking, DP is an information-theoretic notion. As such, it cannot be satisfied with “just” <em>cryptographically</em> secure PRNG. For purists (or complexity theorists) out there, Opacus meets the definition of Computational Differential Privacy. In practice, all randomness available on computers is computational, and I don’t believe drawing the distinction is productive at this point of technology development.</p>
One of the variables needed for gradient computation has been modified by an "inplace" operation Error

Qucheng_Peng: <p>I have met the problem “one of the variables needed for gradient computation has been modified by an “inplace” operation: [torch.floattensor [2048]] is at version 2; expected version 1 instead.” Thanks to all the possible solutions to it, but I just find that they don’t include my situation. I tried the " torch.autograd.set_detect_anomaly(True)", then it told me that “error detected in native batchnorm backward. traceback of forward call that caused the error:”. It did correspond to the layer “torch.nn.BatchNorm1d”, but I just wonder how to set this layer as “inplace” like relu. Thank you for your time and patience!</p>
Changing attributes of dataloader subset modifies entire dataset

enterthevoidf22: <p>i have a dataset object called total_loader<br>
when initialized, it has an attribute called flip transform which is a torch.nn.module object.</p>
<p>i then create train and validation dataloaders by:</p>
<pre><code class="lang-auto">train_loader = Subset(total_loader,train_idx)
val_loader = Subset(total_loader,val_idx)
training_generator = DataLoader(train_loader, **params)
val_generator = DataLoader(val_loader, **params)

</code></pre>
<p>in the dataset object the  <strong>get_item</strong> func checks whether self.flip_transform is not none and if so applies the transform, like this:<br>
in get item…</p>
<pre><code class="lang-auto">if self.flip_transform is not None:
    y = self.flip_transform(y)
    return y
</code></pre>
<p>i want this behavior to happen only during training. i tried to manually set the flip_transform attribute back to none for the validation dataset only, but i see the change in both datasets.</p>
<p>i try something like this:</p>
<pre><code class="lang-auto">val_generator.dataset.dataset.flip_transform = None
</code></pre>
<p>this changes both train and validation generators back to None!</p>
<p>what should i do?</p>

ptrblck: <p>I guess you are changing the attribute of a reference, so you could create different objects for the training and validation datasets instead.<br>
Your code currently doesn’t show the complete workflow, but since <code>total_loader</code> is reused, the same internal dataset would be used.</p>

enterthevoidf22: <p>I was wondering if there’s a way, since I’m implementing my own flip transform class, maybe I can set it such that it won’t run whenever model.eval is called?</p>
Using Apex AMP with PyTorch optimizers causes Attribute Error

seankala: <p>Hi. I’m facing some issues when I’m trying to use PyTorch optimizers with Apex AMP. My environment is:</p>
<ol>
<li>OS: Ubuntu 18.04.5</li>
<li>Python: 3.8.5</li>
<li>PyTorch: 1.7.1</li>
<li>CUDA: 11.0</li>
<li>Apex: 0.9.10.dev0</li>
<li>Transformers: 4.3.3</li>
</ol>
<p>You can reproduce my error with the following code:</p>
<pre><code class="lang-auto">from apex import amp
from transformers import AutoModel
from transformers.optimization import AdamW, get_linear_schedule_with_warmup

model = AutoModel.from_pretrained('bert-base-cased')
model = model.cuda()

new_layer = ["extractor", "bilinear"]
optimizer_grouped_parameters = [{"params": [p for n, p in model.named_parameters() if not any(nd in n for nd in new_layer)], },{"params": [p for n, p in model.named_parameters() if any(nd in n for nd in new_layer)], "lr": 1e-4},]
optimizer = AdamW(optimizer_grouped_parameters, lr=1e-5, eps=1e-10)

model, optimizer = amp.initialize(model, optimizer, opt_level="O1", verbosity=0)

scheduler = get_linear_schedule_with_warmup(optimizer, num_warmup_steps=4000, num_training_steps=10000)
</code></pre>
<p>Commenting out the <code>model, optimizer = amp.initialize</code> line runs the code fine. However, running this script returns the following:</p>
<pre><code class="lang-auto">Traceback (most recent call last):

  File "../train.py", line 33, in finetune
    scheduler = get_linear_schedule_with_warmup(optimizer, num_warmup_steps=warmup_steps, num_training_steps=total_steps)
  File "/home/User/anaconda3/lib/python3.8/site-packages/transformers/optimization.py", line 98, in get_linear_schedule_with_warmup
    return LambdaLR(optimizer, lr_lambda, last_epoch)
  File "/home/User/anaconda3/lib/python3.8/site-packages/torch/optim/lr_scheduler.py", line 205, in __init__
    super(LambdaLR, self).__init__(optimizer, last_epoch, verbose)
  File "/home/User/anaconda3/lib/python3.8/site-packages/torch/optim/lr_scheduler.py", line 74, in __init__
    self.optimizer.step = with_counter(self.optimizer.step)
  File "/home/User/anaconda3/lib/python3.8/site-packages/torch/optim/lr_scheduler.py", line 56, in with_counter
    instance_ref = weakref.ref(method.__self__)
AttributeError: 'function' object has no attribute '__self__'
</code></pre>
<p>What might the issue be, and how could I fix this? Thanks.</p>

ptrblck: <p>We recommend to use the native mixed-precision utility via <a href="https://pytorch.org/docs/stable/amp.html"><code>torch.cuda.amp</code></a> as described <a href="https://discuss.pytorch.org/t/torch-cuda-amp-vs-nvidia-apex/74994/3">here</a>. New features, such as the compatibility with 3rd party repositories (transformers in this case), won’t land in apex/amp, but in native amp instead.</p>
CUDA initialize error

congve1: <p>When I try to use GPU,  I got the error as follow:</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-6-52891be9f0e3&gt; in &lt;module&gt;
----&gt; 1 torch.ones((3,4)).to("cuda")

e:\miniconda3\lib\site-packages\torch\cuda\__init__.py in _lazy_init()
    161             "Cannot re-initialize CUDA in forked subprocess. " + msg)
    162     _check_driver()
--&gt; 163     torch._C._cuda_init()
    164     _cudart = _load_cudart()
    165     _cudart.cudaGetErrorName.restype = ctypes.c_char_p

RuntimeError: CUDA error: unknown error
</code></pre>
<p>This is my environment info.</p>
<pre><code class="lang-auto">PyTorch version: 1.1.0
Is debug build: No
CUDA used to build PyTorch: 10.0

OS: Microsoft Windows 10 Pro
GCC version: Could not collect
CMake version: Could not collect

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: 10.0.130
GPU models and configuration: GPU 0: GeForce GTX 960M
Nvidia driver version: 430.39
cuDNN version: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\bin\cudnn64_7.dll

Versions of relevant libraries:
[pip3] numpy==1.16.3
[conda] blas                      1.0                         mkl
[conda] mkl                       2019.3                      203
[conda] mkl_fft                   1.0.12           py37h14836fe_0
[conda] mkl_random                1.0.2            py37h343c172_0
[conda] pytorch                   1.1.0           py3.7_cuda100_cudnn7_1    pytorch
[conda] torchvision               0.2.2                      py_3    pytorch
</code></pre>
<p>How to fix this error</p>

Deepanshu_Jindal: <p>Did you find any solution to this?<br>
I was able to use the GPU with pytorch a couple of times before this error appeared magically one day.</p>
<p>Edit: I was able to solve the problem using the discussion here <a href="https://github.com/pytorch/pytorch/issues/17108" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/17108</a></p>
<p>What worked for me was calling torch.cuda.current_device() before any cuda calls. For eg.</p>
<pre><code class="lang-auto">import torch
torch.cuda.current_device()
torch.cuda.is_available()
</code></pre>
<p>works but the opposite order of calls will not. Hope it helps!</p>

Jesse_Stone: <p>it’s a version match problem i meet it too. now solved as follows<br>
driver: NVIDIA-Linux-x86_64-440.118.02.run<br>
cuda: cuda_10.2.89_440.33.01_linux.run<br>
cuda patch: cuda_10.2.2_linux.run</p>
<p>libcudnn:<br>
libcudnn7_7.6.5.32-1+cuda10.2_amd64.deb<br>
libcudnn7-dev_7.6.5.32-1+cuda10.2_amd64.deb<br>
libcudnn7-doc_7.6.5.32-1+cuda10.2_amd64.deb</p>
<p>pytorch:<br>
pip3 install torch==1.7.1 torchvision==0.8.2 torchaudio==0.7.2</p>
<p><span class="hashtag">#do</span> not install torch 1.8.0</p>
Error at compile / win10 / _C10_ALWAYS_INLINE / __has_attribute(always_inline)

Sebastien_Maraux: <p>Hello there,</p>
<p>I switched recently a project from libtorch 1.7.0 to 1.8.0, under MS Visual Studio C++ 2017. We are using warning as error in our code, so I had to disable an additional warning 4624 (about automatically deleted destructor or something like that), but also had to patch a c10 macros header file to be able to compile.</p>
<p>The file is :…/libtorch_cuda11_release/include/c10/macros/Macros.h</p>
<p>When declaring _C10_ALWAYS_INLINE , the header uses __has_attribute preprocessor, which does not exist under MSVC.</p>
<p>The code :</p>
<pre><code>#if __has_attribute(always_inline) || defined(__GNUC__)
	#define C10_ALWAYS_INLINE __attribute__((__always_inline__)) inline
#elif defined(_MSC_VER)
	#define C10_ALWAYS_INLINE __forceinline
#else
	#define C10_ALWAYS_INLINE inline
#endif	
</code></pre>
<p>Should be replaced by something like :</p>
<p><span class="hashtag">#if</span> defined __has_attribute<br>
<span class="hashtag">#if</span> __has_attribute(always_inline) || defined(<strong>GNUC</strong>)<br>
<span class="hashtag">#define</span> C10_ALWAYS_INLINE <strong>attribute</strong>((<strong>always_inline</strong>)) inline<br>
<span class="hashtag">#elif</span> defined(_MSC_VER)<br>
<span class="hashtag">#define</span> C10_ALWAYS_INLINE __forceinline<br>
<span class="hashtag">#else</span><br>
<span class="hashtag">#define</span> C10_ALWAYS_INLINE inline<br>
<span class="hashtag">#endif</span>	<br>
<span class="hashtag">#elif</span> defined(_MSC_VER)<br>
<span class="hashtag">#define</span> C10_ALWAYS_INLINE __forceinline<br>
<span class="hashtag">#else</span><br>
<span class="hashtag">#define</span> C10_ALWAYS_INLINE inline<br>
<span class="hashtag">#endif</span></p>
<p>Or anything simpler, testing first for MSVC maybe</p>
<p>Could it be integrated in the nightly ? That would avoid manually patching future versions.</p>
<p>Best</p>

ptrblck: <p>I’m not sure, how well VS2017 is supported, as the <a href="https://github.com/pytorch/pytorch">build instructions</a> mention:</p>
<blockquote>
<p>Visual Studio 2019 version 16.7.6 (MSVC toolchain version 14.27) or higher is recommended.</p>
</blockquote>
<p>CC <a class="mention" href="/u/peterjc123">@peterjc123</a></p>

peterjc123: <p>This is unrelated to VS versions and need a fix. I’ll do that.</p>

peterjc123: <p>Fixed by <a href="https://github.com/pytorch/pytorch/pull/53562" class="inline-onebox" rel="noopener nofollow ugc">[PyTorch] Fix broken build caused by keyword missing on Windows by larryliu0820 · Pull Request #53562 · pytorch/pytorch · GitHub</a>.</p>
nn.MSELoss inputs inquiry

FryingPan: <p>Hello,<br>
I have a question with regards to <code>nn.MSELoss()</code>. I’m trying to implement a network to demosaic/debayer an image. I have seen that the inputs to nn.MSELoss should be <code>nn.MSELoss(outputs, target)</code>. My <code>outputs</code> tensor is of shape <code>[batch_size, 3, 224, 224]</code>, but my target is <code>[batch_size]</code>.</p>
<p>I am loading the images as so.</p>
<pre><code class="lang-auto">trainset = datasets.ImageFolder(root=trainpath,transform=preprocess)
trainloader = DataLoader(dataset=trainset,batch_size=batch_size, shuffle=True)
</code></pre>
<p>And then in my training function</p>
<pre><code class="lang-auto">for data in trainloader:
    inputs, target = data[0], data[1]
    optimizer.zero_grad()
    outputs = model(inputs)
    loss = nn.MSELoss(outputs, target)
</code></pre>
<p>Should it instead be <code>loss = nn.MSELoss(outputs, inputs)</code> as I know MSE subtracts the two values. But from everything I’ve seen, it’s always (outputs, target).</p>
<p>Thank you very much</p>

ptrblck: <p>As you’ve explained, <code>nn.MSELoss</code> expects the model output and target to have the same shape.<br>
Since your target shape differs, you would have to explain a bit more how the loss calculation should be performed for your use case.<br>
E.g. you could repeat the target values for each channel and pixel, but I doubt this would fit your use case.</p>
Does Increasing Image size should be considered a good approach?

Sahil_malik: <p>Hi Learners</p>
<p>Could you pelse guide me about Image resolution importance. I am using transfer learning and some models need minimum resolution for example 96. But my Image sizes are 32. Would it be good approach to resize them?<br>
Secondly, If I meet the maximum reported accuracy after increasing  resolution; would it be still considered as good as the reported max. accuracy with original image sizes ?</p>
<p>Kindly guide me at this matter.</p>
<p>Thank you!</p>
Variable size input for 2D to 1D output

sajidmahmud14: <p>Hello,<br>
So I am trying to implement a network which will take in an L x L input, where L is variable, and output an L sized  1D sequence. Now I know fully convolution networks can handle variable size input. In the network, the L x L goes through several convolutional layers, and then will be fed to an attention layer to output an L sized sequence. However I am not sure how an Attention mechanism can handle a variable size input. One solution I pondered is padding all the inputs to a fixed size, but that would be problematic as the L varies a lot, which might introduce noise and overfitting.</p>
<p>Another solution I pondered is taking mini-batches and padding them to a fixed length, however it seems that at different batches, the network will see inputs with variable L anyway.</p>
<p>What is the most elegant solution to variable size inputs in this case?</p>
Loss bounces around the same no matter what the learning rate is

agt: <p>I am trying to overfit my model to 1 sample, but once the loss gets near 2, it just jumps around between 2 and 4 and doesn’t get better. I thought it was because my learning rate was too high, but no matter how low I set my learning rate, the jumps stay the same:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/f/9f37a0499b471004f2f05a4d2a67e2989c53e398.png" data-download-href="https://discuss.pytorch.org/uploads/default/9f37a0499b471004f2f05a4d2a67e2989c53e398" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/f/9f37a0499b471004f2f05a4d2a67e2989c53e398_2_690x282.png" alt="image" data-base62-sha1="mIv5XS3REJ5I58vH6Jt7R4g2sNq" width="690" height="282" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/f/9f37a0499b471004f2f05a4d2a67e2989c53e398_2_690x282.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/f/9f37a0499b471004f2f05a4d2a67e2989c53e398_2_1035x423.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/f/9f37a0499b471004f2f05a4d2a67e2989c53e398_2_1380x564.png 2x" data-dominant-color="FAFAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1380×564 52.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Based on how SGD/Adam works, I thought it should be guaranteed to find parameters that can overfit 1 sample. What could be causing this?</p>
<p><strong>Details about the model:</strong></p>
<p>I’m implementing this paper <a href="https://arxiv.org/pdf/2002.03230.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/2002.03230.pdf</a>, which involves generating a graph like this: At the first step, select the first component. Then, in every other step: select a new component to attach to an existing component, select the nodes where it can be attached, and finally select the specific pair of nodes where they will be attached.</p>
<p>I make the selections by sampling from the output of softmaxed MLPs. To train, I use NLLLoss with reduction=None, and add <code>self.loss(mlp_log_probs, target_idxs)</code> to a <code>losses</code> list each time I make a selection. At the end, I do <code>sum(losses)</code> and backpropagate.</p>
Memory double when initialize DDP

ConnollyLeon: <p>I found that when I build a model like</p>
<pre><code class="lang-auto">    model = nn.Sequential(*[nn.Linear(2000, 2000).to(rank) for _ in range(20)])
    torch.cuda.synchronize()
    print_peak_memory("Max memory allocated after creating local model", rank)

    # construct DDP model
    ddp_model = DDP(model, device_ids=[rank])
    print_peak_memory("Max memory allocated after creating DDP", rank)
</code></pre>
<p>the memory will definitely double. And this is quite unacceptable when training a large-scale model. Are there any solutions to help with this?</p>
<p>PS: the code is from <a href="https://pytorch.org/tutorials/recipes/zero_redundancy_optimizer.html" rel="noopener nofollow ugc">ZeRO</a></p>

H-Huang: <p>Hello! Yes, the memory required in this example will double when using <a href="https://pytorch.org/docs/master/notes/ddp.html?highlight=distributed%20data%20parallel" rel="noopener nofollow ugc">DDP</a>. This is because the world_size is 2 and the purpose of DDP is for data parallelism (same model, multiple data) where the model architecture is copied across multiple machines or GPUs to handle data in parallel.</p>
<p>In the case of a large-scale model that does not fit on a single machine or GPU, you should look into <a href="https://pytorch.org/docs/master/rpc.html?highlight=rp#module-torch.distributed.rpc" rel="noopener nofollow ugc">Distributed RPC</a>. RPC allows for model parallelism (split model, same data) where you can split the model across machines and use RPC to communicate between the different layers. The RPC framework also handles autograd and optimizer steps internally.</p>
Question about ZeRO optimizer

ConnollyLeon: <p>I’m using ZeroRedundancyOptimizer in torch1.8. And I notice that in the step function of ZeRO, there is a update_param_groups before the self.optim.step().</p>
<p>I wonder if this function broadcast the gradients that the self.optim uses to calculate the new parameters. I don’t see any docstring explains where the gradients go.</p>
<p>Looking forward to any replies, please~~</p>

ConnollyLeon: <p>Ummm, I found the key. DDP has already done the parameters and gradients communications</p>

rvarm1: <p>Exactly, regardless of the optimizer, before <code>step()</code> is called it is guaranteed that all grad communication has taken place and each replica has allreduced gradients computed in the parameter’s .grad fields.</p>
Loss function for an image

Marcel_Iten: <p>What loss functions/ criterion do i choose when my model output is an image? I would like to have pixelwise L2 loss.</p>

ptrblck: <p><code>nn.MSELoss</code> should work in this case and accepts 4-dimensional tensors (image tensors) as the model output and target.</p>

Marcel_Iten: <p>Another question. My network takes the input image and outputs an image of same size. Then i add my input image to the ouput image and that should be equal to the ground image. → Input + Net(Input) = Ground Truth. Do i need to add the input image already in the Net class or can i add the input image to the output by torch.add(input,output) just before calculating the loss?</p>

ptrblck: <p>You can add it directly to the desired tensor and don’t need to do so in the model’s <code>forward</code> method, as there wouldn’t be a difference.</p>
When does one divide by the meta_batch_size for MAML during meta-learning?

Brando_Miranda: <p>I was going through the maml example in the higher library but it seems to me they are forgetting to divide by the <code>meta_batch_size</code>, they have:</p>
<pre><code class="lang-auto">qry_loss.backward()
</code></pre>
<p>where they should have</p>
<pre><code class="lang-auto">meta_batch_size = qry_loss.size(0)
(qry_loss/meta_batch_size).backward()
</code></pre>
<p>is this correct? should they be dividing by the <code>meta_batch_size</code>?</p>
<p>references:</p>
<ul>
<li>full code example working on omniglot: <a href="https://github.com/facebookresearch/higher/blob/master/examples/maml-omniglot.py" class="inline-onebox" rel="noopener nofollow ugc">higher/maml-omniglot.py at master · facebookresearch/higher · GitHub</a>
</li>
<li>crossposted: <a href="https://github.com/facebookresearch/higher/issues/104" class="inline-onebox" rel="noopener nofollow ugc">when do we divide by met_batch_size? · Issue #104 · facebookresearch/higher · GitHub</a>
</li>
</ul>
Code abruptly ends without any reason

arunpatro: <p>Hi,</p>
<pre><code class="lang-auto">model.eval()
with torch.no_grad():
    for idx, (images, targets) in tqdm(enumerate(train_dl), total=len(train_dl)):
        imgs, targs = model.transform(images, targets)
        fv = model.backbone(imgs.tensors.to(device))["0"].cpu()
        feature_vecs.append(fv)
</code></pre>
<p>While executing this code where model is a <code>torchvision.models.detection.faster_rcnn</code> and <code>images</code> and <code>targets</code> are correct.</p>
<p>This loop abruptly stops at 45/1039 in the dataset. I have lots of memory. So why does it randomly terminate?</p>

Dwight_Foster: <p>Does it give an error? Depending on how big feature_vecs gets that could be the problem.</p>
Perceptual loss from discriminator network

terbed: <p>Hi there,</p>
<p>I try to generate audible speech from impaired speech with GANs. There is a constraint that the generated sound should resemble the input (and keep the context): min(input, output). But here I want to use higher-level features utilizing the layers of the discriminator network. I created the discriminator network such that it gives back the intermediate layers too:</p>
<pre><code class="lang-auto">class Discriminator(nn.Module):
    def __init__(self):
        super().__init__()
        .
        .
        .

    def forward(self, x):

        layer_outputs = []

        l1 = self.layer1(x) 
        layer_outputs.append(l1)

        l2 = self.layer2(l1) 
        layer_outputs.append(l2)
        
        l3 = self.layer3(l2)  
        layer_outputs.append(l3)

        l4 = self.layer4(l3)  
        layer_outputs.append(l4)

        l5 = self.layer5(l4) 
        layer_outputs.append(l5)

        out = self.fully(l5)

        return out, layer_outputs
</code></pre>
<p>And there is a function that calculates the sum of the difference of intermediate layers between two inputs:</p>
<pre><code class="lang-auto">    def lap1_loss(D, x: tr.tensor, y: tr.tensor):
        """
        Implements the laplace loss for the discriminator layers.
        Input shape: (N, C, H, W)
        :return: the scalar loss value
        """
        assert x.shape == y.shape, "The shape of inputs must be equal."
        assert len(x.shape) == 4, "Input must be 4 dimensional."

        _, x_acts = D(x)
        _, y_acts = D(y)

        losses = tr.stack([trf.l1_loss(x_l, y_l)*2**(-2*l) for l, (x_l, y_l) in enumerate(zip(x_acts, y_acts))])
        loss = tr.sum(losses)

        return loss
</code></pre>
<p>It seems like it is working, but I am not sure that this is the right way of implementing this. My question is that is this solution correct / is there a more elegant solution?</p>
<p>Thanks!</p>