akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to extract deep features from a pretrained model in pytorch?

DaLa: <p>Hello,<br>
I want to extract deep features from this pretrained model in tensorflow <a href="https://www.kaggle.com/code/amyjang/tensorflow-transfer-learning-melanoma/output" rel="noopener nofollow ugc">https://www.kaggle.com/code/amyjang/tensorflow-transfer-learning-melanoma/output</a>. I tried this but I am not sure if it is right.</p>
<pre><code class="lang-auto">import h5py
file = h5py.File('model_weights.hdf5', 'r')
weights_layer_1 = file['weights/layer_1'][:]
import torch

# create the PyTorch model
model = MyModel()
</code></pre>
<p>Here I need to create the model in pytorch, right?</p>
<p>There is no another way to use to pretrained model without the need to create it first?</p>
<p>Thanks in advance.</p>

fabian_schutze: <p>Hi,</p>
<p>I guess you want to access the features for a particular image / input. If so:</p>
<ul>
<li>you need to know the model of that is associated with the weights (your MyModel)</li>
<li>load the weights into your model.</li>
<li>Pass the input / image trough the backbone</li>
<li>Extract the elements of the backbone at the different stages (for example, the result of self.layer1(x), self.layer2(x), … self.layer4(x)</li>
</ul>
<p>The pytorch tutorials are really nice and should introduce the main concepts for the steps above. If the model in indeed a tensorflow model, you need to work with tensorflow not pytorch.</p>

DaLa: <p>My code is in pytorch and I want to compute perceptual loss by extracting  deep features from this model and add it to my loss function. I think it is not adequate to use pytorch and tensorflow together in the same code.</p>
Map-style data loader for multiple directories/files

Jamie_Donnelly: <p>I’m trying to create a <code>Dataset</code> for a large dataset. The dataset is probably 100-200GB currently partitioned into a few directories and in each directory there is ~50 <code>.pt</code> files each with approx 4,000,000 rows and a variable number of features (each <code>.pt</code> file has a different number of features) and each row of these files is a single sample.</p>
<p>Naturally I can’t and don’t want to hold even a fraction of this in memory at any one time, but I want to iterate through one file at a time because this saves any effort of bucketing batches for training an RNN, as each mini-batch will come from the same file and have the same # of features.</p>
<p>Has anyone encountered this sort of problem before? I thought about storing some sort of internal index that determines which <code>.pt</code> file <code>__getitem__</code> returns from, cacheing the file internally and then when reaching the maximum length initialising a new index and loading the new indexed file to memory,  but I don’t know what impact this will have when using a <code>DistributedSampler</code> in a <code>DataLoader</code> constructor to parallelise training.</p>
<p>I have outlined an un-tested prototype <code>Dataset</code> below</p>
<pre><code class="lang-auto">class FlowData(Dataset):
    def __init__(self,roots):
        super(FlowData,self).__init__()
        self.roots = roots
        self.files = self._files()
        self.N = self._len()
        self._parsed = [] # List of already parsed `.pt` files
        self._global_index = _internal_state # index `.pt` file
        self._local_index = 0 # local index of row in `.pt` file
        self.x, self.y = self._load_internal(self.index)
        pass

    def _load_internal(self,index):
        x, y = torch.load(self.files[self.index])
        return x, y

    def __getitem__(self,index):
        if torch.is_tensor(index):
            index = index.tolist()
        try:
            x, y = self.x[self.local_index], self.y[self.local_index]
            self.local_index += 1
            return x, y
        except IndexError:
            self.local_index = 0
            pass

    def _files(self):
        files = []
        for dir in self.roots:
            for file in os.listdir(dir):
                files.append(os.path.join(dir,file))
        return sorted(files)

    def _len(self):
        N = 0
        for file in self.files:
            data = torch.load(file)
            N += data[0].shape[0]
        return N 
        
    def __len__(self):
        return self.N        
</code></pre>
Optimize_for_mobile double the size of static quantization deeplabv3 image segmentation model

BillyGun27: <pre><code class="lang-auto">import torch
from torch.utils.mobile_optimizer import optimize_for_mobile
from torchvision.models.segmentation import deeplabv3_resnet50


model = deeplabv3_resnet50(pretrained=True)
model.eval()

#apply static quantization
model_to_quantize_qnnpack = copy.deepcopy(model)
qconfig_mapping = get_default_qconfig_mapping("qnnpack")
model_to_quantize_qnnpack.eval()
# prepare
model_prepared_qnnpack = quantize_fx.prepare_fx(model_to_quantize_qnnpack, qconfig_mapping, input_batch)
# calibrate 
calibrate(model_prepared_qnnpack, val_loader)
# quantize
model_quantized_qnnpack = quantize_fx.convert_fx(model_prepared_qnnpack)

scripted_module = torch.jit.script(model_quantized_qnnpack)
# Export lite interpreter version model (compatible with lite interpreter)
scripted_module._save_for_lite_interpreter("model_output/deeplabv3_quant_static_qnnpack_scripted.ptl")

optimized_scripted_module = optimize_for_mobile(scripted_module)
# using optimized lite interpreter model makes inference about 60% faster than the non-optimized lite interpreter model, which is about 6% faster than the non-optimized full jit model
optimized_scripted_module._save_for_lite_interpreter("model_output/deeplabv3_quant_static_qnnpack_optimized.ptl")
</code></pre>
<p>original model: 160MB<br>
quantization model: 40MB<br>
after optimize for mobile: 80MB</p>
<p>for some reason optimize_for_mobile double the size of the model, is this an expected behavior or not?<br>
could someone help me with explanation?</p>
How to used Adam and LBFGS optimizers in one single neural network

Ali_geo: <p>Dear community,</p>
<p>I want to use two optimizers for my model. First I want to use <code>Adam</code> and then <code>LBFGS</code>. This is my code for training the network:</p>
<pre><code class="lang-auto">layers = np.array([2, 20, 20, 1])
PINN = FCN(layers).to(device)
optimizer = torch.optim.LBFGS(PINN.parameters(), max_iter=20, lr=0.001)
def closure():
    optimizer.zero_grad()
    loss = PINN.loss(left_x[idx_l,:], left_y[idx_l,:], right_x[idx_r,:], right_y[idx_r,:], 
                     bottom_x[idx_b,:], bottom_y[idx_b,:], X_train_Nf)
    loss.backward()
    return loss
for i in range(1000):
    loss = optimizer.step(closure)
    with torch.no_grad():
        test_loss_l = PINN.lossBC_l(left_x, left_y.flatten().view(-1,1))
        if (i+1)%250 == 0:
            print('training:', loss.cpu().detach().numpy(), '/ Testing', test_loss_l.cpu().detach().numpy())
</code></pre>
<p>Then, how can I modify it in a way that it uses <code>Adam</code> for training and <code>LBFGS</code> for the test steps.<br>
In advance I very much appreciate any help.</p>
<p>Cheers<br>
Ali</p>
Different output between server and android after using static quantization fx

BillyGun27: <p>I tried static quantization on deeplabv3 image segmentation model</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://pytorch.org/hub/pytorch_vision_deeplabv3_resnet101/">
  <header class="source">
      <img src="https://pytorch.org/favicon.ico?" class="site-icon" width="32" height="32">

      <a href="https://pytorch.org/hub/pytorch_vision_deeplabv3_resnet101/" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>

  <article class="onebox-body">
    <img src="https://pytorch.org/assets/images/pytorch-logo.png" class="thumbnail onebox-avatar" width="500" height="500">

<h3><a href="https://pytorch.org/hub/pytorch_vision_deeplabv3_resnet101/" target="_blank" rel="noopener nofollow ugc">PyTorch</a></h3>

  <p>An open source machine learning framework that accelerates the path from research prototyping to production deployment.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<aside class="onebox githubfolder" data-onebox-src="https://github.com/pytorch/android-demo-app/tree/master/ImageSegmentation">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/pytorch/android-demo-app/tree/master/ImageSegmentation" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h3><a href="https://github.com/pytorch/android-demo-app/tree/master/ImageSegmentation" target="_blank" rel="noopener nofollow ugc">android-demo-app/ImageSegmentation at master · pytorch/android-demo-app</a></h3>

  <p><a href="https://github.com/pytorch/android-demo-app/tree/master/ImageSegmentation" target="_blank" rel="noopener nofollow ugc">master/ImageSegmentation</a></p>

  <p><span class="label1">PyTorch android examples of usage in applications. Contribute to pytorch/android-demo-app development by creating an account on GitHub.</span></p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>the quantization that I use refer to this method<br>
<a href="https://pytorch.org/tutorials/prototype/fx_graph_mode_ptq_static.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/prototype/fx_graph_mode_ptq_static.html</a></p>
<p>the output is similar to before quantization when performed on server, but on android it is defferent.</p>
<p>Does any one idea what cause this?</p>

BillyGun27: <p>After further checking, the output is actually the same, but since in android the output is only one dimension, the order is incorrect and output need to be transposed first to get the correct dimension shape as before quantized.</p>
<p>what make me curious now why the order change? and can it be prevented?</p>
Efficient vectorized implementation of a loop

Alex_Shtof: <p>I have piece of code which is implemented using loops, which I was not able to implement in a vectorized manner.</p>
<pre><code class="lang-auto">for b in range(B):
    for i in range(M - 1):
        for j in range(i + 1, M):
            interaction = (vectors[b, fields[b, i], fields[b, j], :] * vectors[b, fields[b, j], fields[b, i], :]).sum()
            pairwise[b] += interaction
</code></pre>
<p>I tried using gather / scatter, but they support indexing only on one dimension. Is there an efficient way to implement it? <code>fields</code> is a BxM array, <code>vectors</code> is a BxMxMxD array. The result, <code>pairwise</code>, should be an array of size B.</p>
<p>Even without full vectorization, only vectorizing along the B dimension would already be great.</p>
PiPPy's AMP Support

NickGao96: <p>Hi, I’m interested in PiPPy package’s pipeline parallelism capability. As I’m exploring the doc, I couldn’t find instruction on how mixed precision training is supported with PiPPy. Say if I want to train a model in fp16 using pipeline parallelism, how should I do it correctly?</p>

agu: <p>I would expect that AMP would be orthogonal to PiPPy. cc: <a class="mention" href="/u/kwen2501">@kwen2501</a></p>

NickGao96: <p>In theory I would also expect them to be orthogonal. But as PiPPy will do a lot additional things like graph tracing etc, I just wonder how compatible are they in practice?</p>
Swapping in higher dimention

KAISER1997: <p>I have a 3-dimentional pytorch array (rows x Column x K) .Lets name the array as  ‘X’.  I have a list of tuples , ‘L’ . For every tuple (a,b) in ‘L’,   I want to swap the values at X[a,b,a] and X[a,b,b].  Is there a fast pytorch operation for this?</p>
<p>(Here rows=Column=K)</p>

KFrank: <p>Hi Kaiser!</p>
<aside class="quote no-group" data-username="KAISER1997" data-post="1" data-topic="174604" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kaiser1997/48/58253_2.png" class="avatar"> KAISER1997:</div>
<blockquote>
<p>For every tuple (a,b) in ‘L’,   I want to swap the values at X[a,b,a] and X[a,b,b].  Is there a fast pytorch operation for this?</p>
</blockquote>
</aside>
<p>You may use pytorch tensor indexing to both access and set the values of<br>
the desired elements, together with a helper tensor to temporarily store the<br>
values being swapped:</p>
<p>Thus:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print (torch.__version__)
1.13.1
&gt;&gt;&gt;
&gt;&gt;&gt; _ = torch.manual_seed (2023)
&gt;&gt;&gt;
&gt;&gt;&gt; K = 10
&gt;&gt;&gt; N = 5
&gt;&gt;&gt; X = torch.randn (K, K, K)
&gt;&gt;&gt; L = torch.randint (K, (N, 2))
&gt;&gt;&gt;
&gt;&gt;&gt; Xc = X.clone()
&gt;&gt;&gt;
&gt;&gt;&gt; temp = X[L[:, 0], L[:, 1], L[:, 0]]
&gt;&gt;&gt; X[L[:, 0], L[:, 1], L[:, 0]] = X[L[:, 0], L[:, 1], L[:, 1]]
&gt;&gt;&gt; X[L[:, 0], L[:, 1], L[:, 1]] = temp
&gt;&gt;&gt;
&gt;&gt;&gt; L
tensor([[3, 5],
        [7, 6],
        [6, 0],
        [0, 8],
        [5, 9]])
&gt;&gt;&gt; Xc[L[2, 0], L[2, 1]]
tensor([-1.7196, -0.5159, -0.3002,  0.4148, -0.1301, -0.4926,  0.3135, -0.1687,
        -0.5783, -1.2591])
&gt;&gt;&gt; X[L[2, 0], L[2, 1]]
tensor([ 0.3135, -0.5159, -0.3002,  0.4148, -0.1301, -0.4926, -1.7196, -0.1687,
        -0.5783, -1.2591])
</code></pre>
<p>Best.</p>
<p>K. Frank</p>
Error 11: Resource temporarily unavailable

shadow: <p>I am trying to train a neural network under PyTorch. Everything works properly except when i specify in the dataloader num_workers=4 or even 3, it raises the error <strong>Error 11: Resource temporarily unavailable</strong>. I have a decent machine with 128 core CPU and a good gpu as well. I checked with htop to monitor the cpu usage and all the resources seem to be good. Checked with nvidia-smi for the gpu and also, nothing odd to report in terms of memory. Could you please help to troubleshoot this?</p>

ptrblck: <p>Do you see in the stacktrace which call fails and thus which resource might be unavailable?</p>
Cann't add pytorch into cmake project

Abdelkareem_Elkhateb: <p>I am trying to install pytorch with cpp I have spend around 12 hour to get it done but i have a lot of problem my os is pop os when i get the following error:<br>
CMake Error at /home/kemo/Downloads/cpp_libs/libtorch/share/cmake/Caffe2/public/cuda.cmake:185 (message):<br>
PyTorch requires cuDNN 7 and above.<br>
Call Stack (most recent call first):<br>
/home/kemo/Downloads/cpp_libs/libtorch/share/cmake/Caffe2/Caffe2Config.cmake:92 (include)<br>
/home/kemo/Downloads/cpp_libs/libtorch/share/cmake/Torch/TorchConfig.cmake:68 (find_package)<br>
CMakeLists.txt:7 (find_package)</p>
<p>for cuDNN i have installed it and cuda and they are running well !</p>

ptrblck: <p>I guess the error is raised from <a href="https://github.com/pytorch/pytorch/blob/49444c3e546bf240bed24a101e747422d1f8a0ee/cmake/public/cuda.cmake#L185">here</a> which indicates the cuDNN version is older than 7.0.0 as also mentioned in the error message.<br>
Which cuDNN version are you using and does the build log show where cuDNN is found?</p>

Abdelkareem_Elkhateb: <p>This where the cudnn located in my os</p>
<pre><code class="lang-auto">cat /usr/lib/cuda/include/cudnn_version.h | grep CUDNN_MAJOR -A 2
#define CUDNN_MAJOR 8
#define CUDNN_MINOR 1
#define CUDNN_PATCHLEVEL 1
--
#define CUDNN_VERSION (CUDNN_MAJOR * 1000 + CUDNN_MINOR * 100 + CUDNN_PATCHLEVEL)

#endif /* CUDNN_VERSION_H */
</code></pre>
<p>who can i make cmake to find it in this path !<br>
if this is the problem!</p>

Abdelkareem_Elkhateb: <p>This is my cmakeLists file</p>
<pre><code class="lang-auto">cmake_minimum_required(VERSION 3.0 FATAL_ERROR)
project(example-app)


set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${TORCH_CXX_FLAGS}")
set(CMAKE_PREFIX_PATH  "/home/kemo/Downloads/cpp_libs/libtorch/share/cmake/Torch")
find_package(Torch REQUIRED )
add_executable(example-app main.cpp)
target_link_libraries(example-app "${TORCH_LIBRARIES}")
set_property(TARGET example-app PROPERTY CXX_STANDARD 14)
</code></pre>

ptrblck: <p>Does your build process also show the right folders or do you have multiple cuDNN installs?<br>
Does <code>find / -name cudnn_version.h</code> return more than this 8.1.1 file?</p>
ValueError: Expected more than 1 spatial element when training, got input size torch.Size([32, 165, 1, 1])

yoL: <p>ValueError: Expected more than 1 spatial element when training, got input size torch.Size([32, 165, 1, 1])</p>

ptrblck: <p>I don’t know which layer you are using raising this error, but it expects an input with more than a single pixel to e.g. calculate stats of this input activation. Increase the spatial size of your input and it should work.</p>
Q-learning. Can you tell me if the output is correct?

MACHOCAPTCHA: <p>I investigate the error of Q-learning by a function:<br>
‘<strong>MSELoss( q_value, target_q_value)</strong>’<br>
‘<strong>q_target_value</strong>’ I calculate as a copy of ‘<strong>q_value</strong>’ with a change in the cell with the maximum value (cell = action):<br>
<strong>‘target_q_value[action] = (reward + self.gamma * max_next_q_value)’</strong><br>
After I send the value to ‘<strong>MSELoss</strong>’<br>
further as standard.<br>
<strong>'self.optimizer.zero_grad()</strong><br>
<strong>los.backward()</strong><br>
<strong>self.optimizer.step()’</strong><br>
If for example I have an output of six cells:<br>
[5.0, 3.2, 15.2, 9.7, 2.3, 7.7]<br>
Here I have to change the tensor with cell number 2 (15.2)<br>
and I get, for example, such a tensor for ‘t<strong>arget_q_value</strong>’:<br>
[5.0, 3.2, 14.2, 9.7, 2.3, 7.7].<br>
Now, after ‘<strong>MSELoss and backward and optimizer</strong>’ should I get a downgrade in one cell ‘<strong>q_value</strong>’? Or in everyone?<br>
I get a change in all the cells, and they all fall at the same time to a certain point, and after that they don’t move. It seems to me that something is wrong here.</p>
JIT script error when Sequential container takes a Tuple input

qhu: <p>This is a simple net to reproduce my error. I’m passing a Tuple to the <code>forward</code> method and have specified the typing. I think the error is caused by Jit inferring the input type to Sequential’s <code>forward</code> method to be a Tensor, and not a Tuple. How can I fix this error?</p>
<pre><code class="lang-auto">class MyBatchNorm(nn.Module):
    def __init__(self, output_size, d_ids):
        super().__init__()
        self.d_ids = d_ids
        self.net = nn.ModuleDict({f"{d}": nn.BatchNorm1d(output_size) for d in d_ids})
    
    def forward(self, input_tuple: Tuple[torch.Tensor, int]) -&gt; Tuple[torch.Tensor, int]:
        input_tensor, d = input_tuple
        output_tensor = torch.tensor([])
        for d_name, d_norm in self.net.items():
            if f"{d}" == d_name:
                output_tensor = d_norm(input_tensor)
        if len(output_tensor) == 0:
            raise ValueError(f"invalid d {d}, must be {self.d_ids}")
        return output_tensor, d

class MyNet(nn.Module):
    def __init__(self, output_size, d_ids):
        super().__init__()
        dense_layers = [
            MyBatchNorm(output_size, d_ids),
            MyBatchNorm(output_size, d_ids)
        ]
        self.net = torch.nn.Sequential(*dense_layers)
        
    def forward(self, input_tensor: torch.Tensor, d_tensor: torch.Tensor) -&gt; torch.Tensor:
        d = d_tensor.squeeze()[0].item()
        output_tensor, _ = self.net((input_tensor, d))
        return torch.squeeze(output_tensor)
</code></pre>
<p>Error:</p>
<pre><code class="lang-auto">RuntimeError: 

forward(__torch__.___torch_mangle_16.MyBatchNorm self, (Tensor, int) input_tuple) -&gt; ((Tensor, int)):
Expected a value of type 'Tuple[Tensor, int]' for argument 'input_tuple' but instead found type 'Tensor (inferred)'.
Inferred the value for argument 'input_tuple' to be of type 'Tensor' because it was not annotated with an explicit type.
:
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/nn/modules/container.py", line 117
    def forward(self, input):
        for module in self:
            input = module(input)
                    ~~~~~~ &lt;--- HERE
        return input
</code></pre>
Ranking Losses Examples for training a model

Sourabh: <p>Can someone guide me to examples where ranking losses such as “torch.nn.MarginRankingLoss” have been used to perform complete training of a model? I see small examples as on the official page <a href="https://pytorch.org/docs/stable/generated/torch.nn.MarginRankingLoss.html" rel="noopener nofollow ugc">Official MarginRankingLoss page</a>, but I am still wondering how it can be used for complete model training.</p>
<p>If possible I would also like to know if there are other losses specifically made towards ranking and implemented in pytorch.</p>
Is there a benefit to cache images for data loading?

Hui_Liu: <p>Hi,</p>
<p>I see for yolov7 there is the default option to <a href="https://github.com/WongKinYiu/yolov7/blob/main/utils/datasets.py#L450" rel="noopener nofollow ugc">cache all images</a> in memory for faster training.</p>
<p>However, based on <a href="https://discuss.pytorch.org/t/multi-process-data-loading-and-prefetching/98997">this thread</a>,  <code>torch.utils.data.dataloader.DataLoader</code> will, by default, do prefetching? If so, the effect of caching images on training speed would be very minimal? Or I misunderstand how the prefetching works?</p>
<p>Thanks!!</p>

ptrblck: <p>It would depend on the data loading speed and if you would see any bottlenecks or general wait times during the training if the <code>DataLoader</code> is used. If so, creating a cache might give you benefits assuming the cached images avoid the bottleneck in the first place.<br>
However, you would need to check if the cache uses your system RAM (and if you would have enough) or the disk, as the latter might not give you huge benefits.</p>

Hui_Liu: <p>I see. Thanks, Patrick! Yeah, I see slow-up without cache, but my whole dataset is too large to fit into the memory. So I was wondering if I could just prefetch and cache only the next batch. But then I see we already do that in dataloader?</p>

ptrblck: <p>Yes, the <code>DataLoader</code> will prefetch the next batches, but will not cache it.<br>
I don’t know if your Yolo code could cache only a subset of the dataset to avoid running out of memory, but a quick check of the linked code seems to show all samples would be added to the cache.</p>
Use GPU integrated on laptop for deep learning with pytorch

Doduchuy12: <p>I want to train my deep learning model on my laptop with GPU-integrated configuration:</p>
<ul>
<li>AMD Radeon™ Graphics</li>
<li>Display memory: 512MB</li>
<li>Shared memory: 3770MB<br>
I am using lightning-hydra-template for my project. Can I train on my laptop? If if can then how can I do that? Thank you so much!</li>
</ul>

brighteningeyes: <p>if you are using linux, install pytorch with rocm support<br>
if you are on windows which I assume you are, you need to install torch_directml alongside pytorch and get your torch device from there and use it.</p>
Best choice of metric for interest-based age classification

Saswat: <p>I have a dataset which has users (rows) with the list of their interests (IABs), which looks like this</p>
<pre><code class="lang-auto">user_id | gender | list of interests
--------+--------+--------------------------------
user 1  | male   | games, productivity
user 2  | female | games, lifestyle, design
user 3  | male   | travel, games, messaging
user 4  | male   | messaging, blogging, lifestyle
...
</code></pre>
<p>Since the number of unique interests are few (~500) and the number of rows are high (~67M), what feature engineering practices should I follow to get an ML model score a better accuracy?</p>
<p>P.S.: Simple model with one hot/count hot vectorization yields an accuracy of ~52%</p>
Whats the correct max split size to set in google colab when the gpu reserves >> trying to allocate?

Oliver: <p>So I got this error message<br>
<code>If reserved memory is &gt;&gt; allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF</code><br>
Reserved is 37 GB and allocate was 600 MB.</p>
<p>So imho this fits the <code>&gt;&gt;</code> case?</p>
<p>I tried setting to 512 and 256 and 64 but I still get the same error.</p>
<p>Does anyone have experience on how to set this correctly?</p>
<p>Many thanks in advance!</p>
How to choose Learning Rate and Scheduler

imjohnzakkam: <p>I’m using torchvision’s models ResNet18, EfficientNet B0 for training on CIFAR-10, CIFAR-100. The ResNet50 model is converging when I set the learning rate to 0.00025 but, when I change the learning rate to 0.001 or 0.01 the model is not learning at all, i.e; the loss stays constant. I think the issue might be that the gradients might be too huge for backprop. I have also tried to use the Cosine Annealing scheduler and set the weight decay to 1e-4 and 0.05, but still no improvement. Could you please point out where i’m going wrong?</p>

tiramisuNcustard: <p>You didn’t mention which optimizer you used. Here is an article with learning rates used for ResNet18 on CIFAR-10. Perhaps you can use the article to compare your results. <a href="https://wandb.ai/suvadeep/pytorch/reports/Finetuning-of-ResNet-18-on-CIFAR-10-Dataset--VmlldzoxMDE2NjQ1" rel="noopener nofollow ugc">https://wandb.ai/suvadeep/pytorch/reports/Finetuning-of-ResNet-18-on-CIFAR-10-Dataset–VmlldzoxMDE2NjQ1</a></p>

imjohnzakkam: <p>Hi, Thanks for your reply. Could you please share the code which you used for your runs? Thanks!</p>

tiramisuNcustard: <p>Here is a kaggle notebook: <a href="https://www.kaggle.com/code/greatcodes/pytorch-cnn-resnet18-cifar10" rel="noopener nofollow ugc">https://www.kaggle.com/code/greatcodes/pytorch-cnn-resnet18-cifar10</a></p>
<p>Optimizer and scheduler are defined here in the same notebook: <a href="https://www.kaggle.com/code/greatcodes/pytorch-cnn-resnet18-cifar10?scriptVersionId=42951150&amp;cellId=12" rel="noopener nofollow ugc">https://www.kaggle.com/code/greatcodes/pytorch-cnn-resnet18-cifar10?scriptVersionId=42951150&amp;cellId=12</a></p>
Pull Request closed after approval

Atharva: <p>Hi,</p>
<p>I contributed a small change in the PyTorch documentation. But the related pull request was closed even after approval. Since this is my first time contributing to open source, I expected the PR to be merged after approval and I will receive my first Pair Extraordinaire badge <img src="https://discuss.pytorch.org/images/emoji/apple/hatching_chick.png?v=12" title=":hatching_chick:" class="emoji" alt=":hatching_chick:" loading="lazy" width="20" height="20">. Can someone please explain why the PR was ‘closed’ instead of ‘merged’?</p>
<p>Link to the pull request: <a href="https://github.com/pytorch/pytorch/pull/93014" class="inline-onebox" rel="noopener nofollow ugc">Corrected grammar in contribution guide by atharvakavitkar · Pull Request #93014 · pytorch/pytorch · GitHub</a></p>
<p>Thanks!</p>

ptrblck: <p>Your PR was merged via:</p>
<pre><code class="lang-auto">@pytorchbot merge -f "Small doc grammar fix"
</code></pre>
<p>in <a href="https://github.com/pytorch/pytorch/commit/627282fa6c22e470a4db14e7782bf53335b396be">this commit</a>.</p>

Atharva: <p>Thanks for the prompt response, but then why is the PR status ‘closed’ and not ‘merged’?</p>
The type of input of DataParallel

jiang_ix: <p>Hi, I have a question. For the torch.nn.DataParallel, I use the following code:</p>
<pre><code class="lang-auto">model = DataParallel(MyModule1()).to('cuda:0')
y = model(x)
</code></pre>
<p>Does the type of x must be torch.Tensor? Can I use other types such as list, tuple …?<br>
Thanks a lot</p>

JuanFMontesinos: <p>You can use whatever your forward function deals with</p>
<p>Just to clarify. Native nn.Modules such as convolutions fully conected etcetera do require a torch tensor.<br>
If you create your own nn-Module, you will have to code its forward function.<br>
Inside this function you can do “whatever” you want. If you code it assuming your input is a list of tensors then it’s fine. Remeber python lists are really slow to be dealt with if you use loops. Try not to use them</p>

fabian_schutze: <p>How does the data sharding work with a dict input?</p>
<p>A good example for using only a dict as input might be the base detector of detectron2: <a href="https://github.com/facebookresearch/detectron2/blob/main/detectron2/modeling/meta_arch/dense_detector.py#L75" class="inline-onebox" rel="noopener nofollow ugc">detectron2/dense_detector.py at main · facebookresearch/detectron2 · GitHub</a></p>
<p>According to the docu, dicts will be shallow copied. How does the model replica X know on which subset of the data to operate?</p>
How to solve error

Ayman_Abdelsattar_Me: <p>def imshow(image, ax=None, title=None):<br>
“”“Imshow for Tensor.”“”<br>
if ax is None:<br>
fig, ax = plt.subplots()</p>
<pre><code># PyTorch tensors assume the color channel is the first dimension
# but matplotlib assumes is the third dimension
image = np.array(image)
image = image.numpy().transpose((1, 2, 0))

# Undo preprocessing
mean = np.array([0.485, 0.456, 0.406])
std = np.array([0.229, 0.224, 0.225])
image = (std * image) + mean

# Image needs to be clipped between 0 and 1 or it looks like noise when displayed
image = np.clip(image, 0, 1)
 
ax.imshow(image)

return ax
</code></pre>
<p>image_path = ‘flowers/train/10/image_07086.jpg’<br>
img = process_image(image_path)<br>
<span class="hashtag">#img</span>.shape<br>
imshow(img)<br>
#(np.array([0.229, 0.224, 0.225])).shape<br>
<span class="hashtag">#img</span> = img.transpose ((1,2,0))<br>
<span class="hashtag">#img</span></p>
<hr>
<p>AttributeError                            Traceback (most recent call last)<br>
 in ()<br>
24 img = process_image(image_path)<br>
25 <span class="hashtag">#img</span>.shape<br>
—&gt; 26 imshow(img)<br>
27 #(np.array([0.229, 0.224, 0.225])).shape<br>
28 <span class="hashtag">#img</span> = img.transpose ((1,2,0))</p>
<p> in imshow(image, ax, title)<br>
7     # but matplotlib assumes is the third dimension<br>
8     image = np.array(image)<br>
----&gt; 9     image = image.numpy().transpose((1, 2, 0))<br>
10<br>
11     # Undo preprocessing</p>
<p>AttributeError: ‘numpy.ndarray’ object has no attribute ‘numpy’</p>

marksaroufim: <p>Issue is probably in this line <code>image = image.numpy().transpose((1, 2, 0))</code> you need to remove the <code>.numpy()</code> call because the image is already a numpy array and numpy arrays don’t have a <code>numpy()</code> function since they don’t need one. Torch tensors do have it though</p>

Ayman_Abdelsattar_Me: <p>many thanks for your support</p>
Is pytorch-cuda going to be registered on conda-forge?

hiroalchem: <p>Currently pytorch and torchvision are in anaconda’s conda-forge channel, but pytorch-cuda is not, so I cannot complete the install in the conda-forge channel. This is a problem under certain circumstances. Is there any plan to register pytorch-cuda in conda-forge?</p>

ptrblck: <p>I don’t believe the <code>conda-forge</code> copies of PyTorch, <code>torchvision</code>, or another PyTorch-related library are uploaded by us, but it seems by the <code>conda-forge</code> community/maintainers.<br>
My guess is they might also clone new dependencies to <code>conda-forge</code> once the next stable release is out.</p>
<p>Btw. when I check <a href="https://anaconda.org/conda-forge/pytorch/files"><code>https://anaconda.org/conda-forge/pytorch/files</code></a> it seems CPU-only binaries were cloned?<br>
If so, then I don’t think <code>pytorch-cuda</code> would even be needed or are the CUDA-enabled binaries stored somewhere else on <code>conda-forge</code>?</p>

hiroalchem: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a>.<br>
I understand the situation now.<br>
So then, in order to get the gpu version pytorch on conda-forge, I would just have to register a copy of the gpu version pytorch on the pytorch channel on the conda-forge channel.<br>
What is the procedure to do this? Can I proceed on my own? (Can I do it?)</p>

ptrblck: <p>I don’t know how the <code>conda-forge</code> upload works and who is doing it, but why do you want to clone the binaries inste as d of using the already uploaded ones?</p>

hiroalchem: <p>Thanks for the comment back, it appears that uploads in conda-forge are not restricted to the original library administrator or otherwise restricted. I am checking to make sure it is consistent with some pytorch-related ones that already exist.<br>
To be more clear, I am developing a plugin for a viewer library called napari, and although the napari gui allows the library (plugin) to be installed via conda, I cannot specify any channel other than conda-forge, so I cannot install the library from the pytorch channel.</p>

hiroalchem: <p>By the way, what is the actual entity of pytorch-cuda, shouldn’t it be pytorch + cudatoolkit?</p>

ptrblck: <p><code>pytorch-cuda</code> is a metapackage specifying dependencies. What do you mean by “entity”?</p>

hiroalchem: <blockquote>
<p><code>pytorch-cuda</code> is a metapackage specifying dependencies.</p>
</blockquote>
<p>Sorry, my words were not clear, but that is exactly what I wanted to know.</p>
Training hangs on multi-gpu during loss.backward()

KelS: <p>Hi all,</p>
<p>I am very much in distress because I have been searching on forums and github issues for solution but still haven’t come across any. I am using a machine with 2x RTX 2080TI. However, when I run my set of codes for multi-gpu using nn.DataParallel, it hangs on the loss.backwards() line of code. It runs fine on single-gpu though, without any issues. This loss function is custom and works as issued on the author’s github page.</p>
<p>I have another set of codes(different work) that can run on both multi-gpu and single-gpu without issues. The difference between this two set of codes are, the one with problems in multi-gpu is actually a custom focal loss nn module, where as the other set of codes which works both in single and multi gpu is based on NN.BCELoss().</p>
<p>May I know what’s going on? What should I look into to diagnose this problem? I am really really clueless and in a lot of anxiety due to deadlines coming up.</p>
<p>Thank you.</p>

KelS: <p>Hi all,</p>
<p>I’ve tried running the original codes that utilizes multiprocessing functions on the multi-gpu setup and it works perfectly fine. In my modified set of codes, I’ve modified it to not go through multiprocessing (as I was originally based on Windows, then i shifted to Ubuntu).</p>
<p>May I know if there’s any modules that are required when running on ubuntu? I am very new to ubuntu and I am clueless to where I should start when programming deep learning models.</p>
<p>Thank you.</p>

winchest: <aside class="quote no-group" data-username="KelS" data-post="1" data-topic="97361">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kels/48/13192_2.png" class="avatar"> KelS:</div>
<blockquote>
<p>custom focal loss nn module</p>
</blockquote>
</aside>
<p>HI! I had the same problem <img src="https://discuss.pytorch.org/images/emoji/apple/sob.png?v=12" title=":sob:" class="emoji" alt=":sob:" loading="lazy" width="20" height="20"> Did u fix it?</p>

KelS: <p>Hi, I am no longer in this field of work but from what I remember, the two GPUs that I had wasn’t setup properly. What I did was to go through detailed tutorials on setting up a dual GPU setup on <em>Ubuntu</em>.</p>
<p>Sorry if this isn’t much of a help. All the best <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>
__all__ += [name for name in dir(_C) NameError: name '_C' is not defined pytorch

RodolfoFR: <p>I’m making an application where I use torch, but when I run the file importing torch it always gives this error:</p>
<p><strong>all</strong> += [name for name in dir(_C)<br>
NameError: name ‘_C’ is not defined</p>
<p>does anyone know how to solve?</p>
<p>what i know _C is a folder inside of torch folder</p>

ptrblck: <p>I’m not such which methods you are looking for but maybe importing <code>torch._C</code> would work for you.</p>

SPWUSY: <p>Have you solved your problem？</p>
Pytorch Transformer Tutorial - What is tgt_input = tgt[:-1, :] supposed to do?

Theo: <p><a href="https://pytorch.org/tutorials/beginner/translation_transformer.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/translation_transformer.html</a></p>
<p>The training loop in the above link:</p>
<pre><code class="lang-auto">from torch.utils.data import DataLoader

def train_epoch(model, optimizer):
    model.train()
    losses = 0
    train_iter = Multi30k(split='train', language_pair=(SRC_LANGUAGE, TGT_LANGUAGE))
    train_dataloader = DataLoader(train_iter, batch_size=BATCH_SIZE, collate_fn=collate_fn)

    for src, tgt in train_dataloader:
        src = src.to(DEVICE)
        tgt = tgt.to(DEVICE)

        tgt_input = tgt[:-1, :]

        src_mask, tgt_mask, src_padding_mask, tgt_padding_mask = create_mask(src, tgt_input)

        logits = model(src, tgt_input, src_mask, tgt_mask,src_padding_mask, tgt_padding_mask, src_padding_mask)

        optimizer.zero_grad()

        tgt_out = tgt[1:, :]
        loss = loss_fn(logits.reshape(-1, logits.shape[-1]), tgt_out.reshape(-1))
        loss.backward()

        optimizer.step()
        losses += loss.item()

    return losses / len(list(train_dataloader))
</code></pre>
<p>I dont understand what exactly the parts <code>tgt_input = tgt[:-1, :]</code> and <code>tgt_out = tgt[1:, :]</code> achieve.</p>
<p>It seems to take in tgt as input excluding the last batch, and then using tgt again for the labels but excluding the first batch? If this is what it does, then what purpose does it serve?</p>

Theo: <p>After some digging, it turns out that those were for shifting the target tokens. During training, the outputs that are given to the decoder part of the transformer as input are shifted left.</p>
<p>decoder input: <code>tgt_input = tgt[:-1, :]</code> → shifts the decoder input to the left.<br>
labels: <code>tgt_out = tgt[1:, :} </code> → shifts the output to the right for calculating the loss.</p>
<p>However, it was not clear from the tutorial that the batch dimension is the second dimension of <code>tgt</code>, which was what confused me. In my dataset, I have <code>tgt</code> as <code>tgt.shape = [ batch_dim, sequence_length ]</code>, and the 3rd dimension is for embedding like this: <code>[ batch_dim, seq_len, embedding_dim ]</code>. So when I followed the tutorials training function, it was shifting the batches, not the tokens, which made 0 sense.</p>
<p>I hope this helps if someone else gets confused about it.</p>