akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Build failed with TBB

Vlad_Dumitru: <p>I am trying to build PyTorch from source using TBB instead of OpenMP but I am getting the following errors every time:</p>
<pre><code class="lang-auto">    /usr/bin/ld: &lt;PATH&gt;/pytorch/build/lib/libtorch_cpu.so: undefined reference to `omp_get_max_threads'
    /usr/bin/ld: &lt;PATH&gt;/pytorch/build/lib/libtorch_cpu.so: undefined reference to `omp_get_num_threads'
    /usr/bin/ld: &lt;PATH&gt;/pytorch/build/lib/libtorch_cpu.so: undefined reference to `GOMP_barrier'
    /usr/bin/ld: &lt;PATH&gt;/pytorch/build/lib/libtorch_cpu.so: undefined reference to `GOMP_parallel'
    /usr/bin/ld: &lt;PATH&gt;/pytorch/build/lib/libtorch_cpu.so: undefined reference to `omp_in_parallel'
    /usr/bin/ld: &lt;PATH&gt;/pytorch/build/lib/libtorch_cpu.so: undefined reference to `omp_get_thread_num'
    collect2: error: ld returned 1 exit status
    make[2]: *** [caffe2/CMakeFiles/kernel_function_legacy_test.dir/build.make:109: bin/kernel_function_legacy_test] Error 1
</code></pre>
<p>You can find the full log file <a href="https://drive.google.com/file/d/1YXSicCUcDm2jZyY8n0al8EdJ9TbychWO/view?usp=sharing" rel="noopener nofollow ugc">HERE</a>:</p>
<h2>
<a name="to-reproduce-1" class="anchor" href="#to-reproduce-1"></a>To Reproduce</h2>
<p>Steps to reproduce the behavior:</p>
<ol>
<li><code>git clone https://github.com/pytorch/pytorch</code></li>
<li><code>cd pytorch</code></li>
<li><code>ATEN_THREADING=TBB BUILD_BINARY=1 USE_EIGEN_THREADPOOL=1 USE_CUDA=0 PARALLEL_BACKEND=NATIVE_TBB USE_OPENMP=0 USE_TBB=1 MKL_THREADING=TBB BLAS=MKL USE_MKLDNN=1 MKLDNN_THREADING=TBB BUILD_BINARY=1 python setup.py build --cmake 2&gt;&amp;1 | tee ~/output.txt </code></li>
</ol>
<ul>
<li>PyTorch Version (e.g., 1.0): 1.10.0a0+git1798ff0</li>
<li>OS (e.g., Linux): Manjaro</li>
<li>How you installed PyTorch (<code>conda</code>, <code>pip</code>, source): pip</li>
<li>Python version: 3.9.5</li>
<li>CUDA/cuDNN version: NO</li>
<li>GPU models and configuration: NO</li>
</ul>

Jackson_Scott: <p>Did you solve this issue? I am running into the same thing but with 1.5.1</p>

Vlad_Dumitru: <p>Yes I did, so far as I understood it was because of some flags for whatever reason. If I am running the fallowing command everything is working properly.</p>
<p><code>BUILD_BINARY=1 USE_EIGEN_THREADPOOL=1 USE_CUDA=0 PARALLEL_BACKEND=NATIVE_TBB USE_OPENMP=0 USE_TBB=1 MKL_THREADING=TBB BLAS=MKL USE_MKLDNN=1 MKLDNN_THREADING=TBB BUILD_BINARY=1 python setup.py build --cmake 2&gt;&amp;1 | tee ~/output.txt </code></p>
Define model components when building computation graph?

kyy: <p>Hi, I noticed that when we define a pytorch model, we usually need to specify its components before applying <code>forward()</code> function.  An example is this:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = torch.flatten(x, 1) # flatten all dimensions except batch
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

</code></pre>
<p>here we need to defined <code>self.conv1</code>, <code>self.conv2</code>, etc…  While in keras, we usually directly define these components as we build the computation graph, e.g.</p>
<pre><code class="lang-auto">encoder_input = keras.Input(shape=(28, 28, 1), name="img")
x = layers.Conv2D(16, 3, activation="relu")(encoder_input)
x = layers.Conv2D(32, 3, activation="relu")(x)
x = layers.MaxPooling2D(3)(x)
x = layers.Conv2D(32, 3, activation="relu")(x)
x = layers.Conv2D(16, 3, activation="relu")(x)
</code></pre>
<p>Here we do not need to define several <code>model.Conv2D</code> in advance, but instead define them as we build the graph.</p>
<p>I am wondering if there are similar way to do this in PyTorch?  I am asking this since:</p>
<ol>
<li>sometimes it could be error-prone to keep track of what components I have defined in <code>__init__()</code> and what components are being used in <code>forward()</code>
</li>
<li>when the number of layers are large, it may not be quite convenient to label all layers with unique index.</li>
</ol>
<p>If anyone has idea, please let me know, thanks!</p>
Concurrent evaluations using same model on a GPU in C++/CUDA

R90: <p>I have been working on a project recently that uses CUDA functions to perform many evaluations of the <em>same</em> deep neural network model in parallel across many threads of a GPU in a control optimization problem in a C++/CUDA environment. In this setup, each thread gets a different input vector and access to a shared memory array storing the model parameters. The current implementation is very low-level and effectively amounts to performing the matrix linear combination and non-linear activation operations of the neural network using custom CUDA functions. Because of these custom functions, we’re currently limited to standard multilayer perceptron models. We would like to expand our system’s capabilities and incorporate other architectures such as CNNs and RNNs.</p>
<p>Does PyTorch support something like this in a C++/CUDA environment (TorchScript?) where I can load a model of any architecture with over to the GPU and use PyTorch’s math library to perform many forward-pass model evaluations in parallel using different CUDA threads each with a different input vector? I haven’t come across anything online that matches what I’m looking for, but I may just not be searching for the correct thing with the right terminology.</p>
<p>Hopefully this is clear enough to get across what I’m currently doing and looking for. Let me know if I need to clarify to elaborate more on what I’m trying to do.</p>
Recommendations for hyperparameter optimization packages

keldysh: <p>I’m trying to solve a regression problem (~10 inputs, 1 output) and have based my code on curiosily’s <a href="https://curiousily.com/posts/build-your-first-neural-network-with-pytorch/" rel="noopener nofollow ugc">tutorial</a> (even though it’s for binary classification, it should work for regression after minor tweaking).</p>
<p>Now I’m looking to optimize the hyperparameters of my NN. I want some algorithm to help me optimize the:</p>
<ul>
<li>learning rate and momentum</li>
<li>no. hidden layers, no. hidden units in each layer</li>
<li>learning rate decay</li>
<li>mini-batch size</li>
</ul>
<p>I’m aware of optuna, ax, hyperopt, and ray. There could be others, but I’m keen to see if there are any packages particularly well suited to solving regression problems (and have good docs/tutorials/example code).</p>
Use of DistributedDataParallel

oasjd7: <pre><code class="lang-auto">parser.add_argument('--dist_url', default="env://", type=str)
parser.add_argument('--rank', type=int)
parser.add_argument('--gpu_to_work_on', type=int)
params = parser.parse_args()

def example():
    from torch.nn.parallel import DistributedDataParallel as DDP

    init_dist(args)

    model = nn.Linear(10, 10).cuda(params.gpu_to_work_on)
    ddp_model = DDP(model, device_ids=[params.gpu_to_work_on])

    loss_fn = nn.MSELoss()
    optimizer = optim.SGD(ddp_model.parameters(), lr=0.001)

    outputs = ddp_model(torch.randn(20, 10).cuda(params.gpu_to_work_on))
    labels = torch.randn(20, 10).cuda(params.gpu_to_work_on)

    loss_fn(outputs, labels).backward()
    optimizer.step()


def init_dist(params):
    params.rank = int(os.environ["RANK"])
    params.world_size = int(os.environ["WORLD_SIZE"])
    dist.init_process_group(
        backend="nccl",
        init_method=params.dist_url,
        world_size=params.world_size,
        rank=params.rank,
    )
    params.gpu_to_work_on = params.rank % torch.cuda.device_count()
    print('rank:', params.rank)
    print('gpu_to_work_on:', params.gpu_to_work_on)
    print('n_gpus:', torch.cuda.device_count())
    torch.cuda.set_device(params.gpu_to_work_on)
    return

if __name__ == '__main__':
    example()

</code></pre>
<p><code>python -m torch.distributed.launch main.py</code><br>
Am I correct using <code>DistributedDataParallel</code>?</p>

gcramer23: <p>Hi <a href="https://discuss.pytorch.org/u/oasjd7">oasjd7</a>. It looks like you have done what’s required to use DDP. You have initialized the process group and created a DDP model. You are using the torch.distributed.launch to run your example. Are you having problems with your example code?</p>

oasjd7: <p><a class="mention" href="/u/gcramer23">@gcramer23</a>  Thanks for your reply. There was no error but, I just wondered that my code is working fine as I intended.</p>
<p>I use only 32 batch_size and I don’t need to size up the batch. I just want to more speed. In this situation, does  <code>DistributedDataParallel</code> can help me to speed up? (I have 4-gpus 12,000MiB per gpu, and my code needs at least 40,000MiB)</p>

gcramer23: <blockquote>
<p>I use only 32 batch_size and I don’t need to size up the batch. I just want to more speed. In this situation, does <code>DistributedDataParallel</code> can help me to speed up? (I have 4-gpus 12,000MiB per gpu, and my code needs at least 40,000MiB)</p>
</blockquote>
<p>Yes DDP can improve training speed. It will need to be configured correctly. You can benchmark your configurations to see what works best for your use case.</p>
<p>This paper provides helpful information on this topic <a href="https://research.fb.com/wp-content/uploads/2020/08/PyTorch-Distributed-Experiences-on-Accelerating-Data-Parallel-Training.pdf" class="inline-onebox" rel="noopener nofollow ugc">Log into Facebook</a>.</p>
Cloning parameters slow compared to copying

Joe_Harrison: <p>I am using Libtorch for a genetic algorithm where I need to clone individuals and mutate the clones. I have written a method to do this. This method is one of the most used functions in my code and I have found that removing my clone operation in favour of a copy operation speeds up the code almost 5x.</p>
<p>So I think I might be doing something very wrong. I cannot use a simple copy operation as the changes in copied individuals will reflect on the original. I’ve added a small toy example to show-case this issue:</p>
<pre><code class="lang-auto">    struct NodeTest : torch::nn::Module {
public:
    NodeTest(){
        value = register_parameter("val", torch::rand({1}, torch::TensorOptions().dtype(torch::kFloat)).requires_grad_(true));
    }

    void change_value(){
        this-&gt;parameters()[0].data() = torch::rand({1}, torch::TensorOptions().dtype(torch::kFloat));
    }

    torch::Tensor value;
};

struct IndividualTest {
public:
    IndividualTest() {
        input_nodes.emplace_back(NodeTest());
    }

    IndividualTest clone(){
        IndividualTest new_ind = IndividualTest();
        new_ind = *this;

        new_ind.input_nodes[0] = NodeTest();
        new_ind.input_nodes[0].parameters()[0].data() = this-&gt;input_nodes[0].parameters()[0].data();

        return new_ind;
    }

    void print(){
        cout&lt;&lt;this-&gt;input_nodes[0].parameters()[0].data().item&lt;float&gt;()&lt;&lt;endl;
    }

    void change_value(){
        this-&gt;input_nodes[0].change_value();
    }

    std::vector&lt;NodeTest&gt; input_nodes;
};

    IndividualTest original = IndividualTest();
    IndividualTest copy = original;

    cout&lt;&lt;"original before: "&lt;&lt;endl;
    original.print();

    copy.change_value();

    cout&lt;&lt;"copy: "&lt;&lt;endl;
    copy.print();

    cout&lt;&lt;"original after: "&lt;&lt;endl;
    original.print();

    IndividualTest clone = original.clone();

    cout&lt;&lt;"#######new test#########"&lt;&lt;endl;

    cout&lt;&lt;"original before: "&lt;&lt;endl;
    original.print();

    clone.change_value();

    cout&lt;&lt;"clone: "&lt;&lt;endl;
    clone.print();

    cout&lt;&lt;"original after: "&lt;&lt;endl;
    original.print();
</code></pre>
<p>In my actual code an individual contains many nodes, but the node structure is approximately the same. How can I avoid this costly cloning operation? The registered parameters all have the same memory address when I copy them (like a shared pointer?). I need them to be unique for each individual</p>

ptrblck: <p>The difference between the <code>clone</code> and <code>copy</code> in your code would be that the former allocates new memory for the object (also known as a <code>deepcopy</code> in Python) while the latter would share the same parameters/memory and would thus be faster. If you need to create a clone of the model I don’t think there is a way around it.</p>
The issues of running Pytorch tutorial on customizing dataloader class

Huaiyang_Gongzi: <p>Hello, I was following the Pytorch tutorial of [customizing dataloader class], everything works just fine except when running the last segment of code. It just stay there without finishing, and does not print any output either.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/9/094698ffbc4883c30f2a90af0a3ce1d3a2a77bf4.png" data-download-href="https://discuss.pytorch.org/uploads/default/094698ffbc4883c30f2a90af0a3ce1d3a2a77bf4" title="tutorialoutput"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/9/094698ffbc4883c30f2a90af0a3ce1d3a2a77bf4_2_568x500.png" alt="tutorialoutput" data-base62-sha1="1k3ydUz0l0cWA4fkF0n7dLVzsd6" width="568" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/9/094698ffbc4883c30f2a90af0a3ce1d3a2a77bf4_2_568x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/9/094698ffbc4883c30f2a90af0a3ce1d3a2a77bf4_2_852x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/9/094698ffbc4883c30f2a90af0a3ce1d3a2a77bf4_2_1136x1000.png 2x" data-dominant-color="EFEFEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">tutorialoutput</span><span class="informations">1137×1000 95.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>(<a href="https://pytorch.org/tutorials/recipes/recipes/custom_dataset_transforms_loader.html" class="inline-onebox" rel="noopener nofollow ugc">Developing Custom PyTorch Dataloaders — PyTorch Tutorials 1.7.1 documentation</a>)</p>

ptrblck: <p>Could you set the <code>num_workers</code> to <code>0</code> and try to rerun the cell?<br>
Depending on your setup etc. multiprocessing might run into issues (e.g. the <a href="https://pytorch.org/docs/stable/notes/windows.html">Windows FAQ</a> explains the if-clause guard, which is needed for Windows setups).</p>

Huaiyang_Gongzi: <p>Hi ptrbick, thanks for the response. It works after reducing the num_workers to zero. So another approach is to add if-then clause. I will give it a try later.</p>
Key-value store shared memory with tensors

theoryofjake: <p>Hey y’all,</p>
<p>I had a quick question about making a distributed pytorch application I had built more efficient. Basically, I am implementing a full-torch version of Ape-X (<a href="https://arxiv.org/pdf/1803.00933.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1803.00933.pdf</a>). In it, the authors implement a shared replay buffer in shared memory with a tensorflow key-value store (Appendix F).</p>
<p>My issue occurs that, since I can’t really specify how much compute each actor uses (I’m assuming one device per each item in world_size), my CPUs just get destroyed with handling all these RPCs, and my hope was to build this system to scale well (independent of the number of actors).</p>
<p>My question–are there ops such as tensorflow’s <a href="https://www.tensorflow.org/api_docs/python/tf/lookup" rel="noopener nofollow ugc">lookup</a> module for this stuff? Can I put tensors in shared memory? I understand the TCPStore only takes strings in its set() args. Is there a torch recommendation for handling the case where I have centralized data that needs to get read by a learner and added to by a bunch of actors generating data, without requests overload?</p>
<p>Thanks for any time y’all.</p>

gcramer23: <p>Hi <a href="https://discuss.pytorch.org/u/theoryofjake">theoryofjake</a>,</p>
<blockquote>
<p>Can I put tensors in shared memory?</p>
</blockquote>
<p>You can move tensors to shared memory <a href="https://pytorch.org/docs/stable/generated/torch.Tensor.share_memory_.html" class="inline-onebox" rel="noopener nofollow ugc">torch.Tensor.share_memory_ — PyTorch 2.1 documentation</a>.</p>
<p>A possible solution is to use a python dictionary and move the values to shared memory. Will this work for your case?</p>

theoryofjake: <p>Could you give an example? My thought was something similar to using torch.distributed’s TCPStore, something like</p>
<pre><code class="lang-auto">store = torch.distributed.TCPStore(..)
rand_tensor = torch.rand((64, 10))
store.set('rand_tensor1', rand_tensor)
</code></pre>
<p>but this errors because set() only takes str as the second arg.</p>

gcramer23: <p>example using tensor.share_memory_()</p>
<pre><code class="lang-auto">import torch
import torch.multiprocessing as mp

def run(rank, python_dict):
    python_dict[rank] += torch.randn(2)

def run_example(share_memory):
    print(f"share_memory={share_memory}")
    nproc=4
    python_dict = {}
    for i in range(nproc):
        python_dict[i] = torch.zeros(2)
        if share_memory:
            python_dict[i].share_memory_()
    print(f"before={python_dict}")
    processes = []
    for rank in range(nproc):
        p = mp.Process(target=run, args=(rank, python_dict,))
        processes.append(p)
    for proc in processes:
        proc.start()
        proc.join()
    print(f"after={python_dict}\n")

if __name__ == "__main__":
    run_example(share_memory=False)
    run_example(share_memory=True)


</code></pre>
Can transforms.Compose handle a batch of Images?

timbmg: <p>I have a preprocessing pipeling with <code>transforms.Compose()</code>. However, I’m wondering if this can also handle batches in the same way as <code>nn.Sequential()</code> ?</p>
<p>A minimal example, where the img_batch creation doesn’t work obviously…</p>
<pre><code>import torch
from torchvision import transforms
from PIL import Image

img1 = Image.open('img1')
img2 = Image.open('img2')
img3 = Image.open('img3')

img_batch = torch.Tensor([img1, img2, img3])

preproc = transforms.Compose([
    transforms.ToTensor()
])

preproc(img_batch)</code></pre>

fabioperez: <p>Check this discussion: <a href="https://github.com/pytorch/vision/issues/157" rel="nofollow noopener">https://github.com/pytorch/vision/issues/157</a>.</p>

Eric_Hou: <p>thanks, so the simple answer is no. <img src="https://discuss.pytorch.org/images/emoji/apple/neutral_face.png?v=9" title=":neutral_face:" class="emoji" alt=":neutral_face:"></p>

JosueCom: <p>3 years later … haha</p>

ptrblck: <p>Yes, it can, if you pass tensors to it:</p>
<pre><code class="lang-python">transform = transforms.Compose([
    transforms.RandomCrop(24),
    transforms.Normalize([0.5, 0.5, 0.5], [0.5, 0.5, 0.5])
])

x = torch.randn(64, 3, 30, 30)
out = transform(x)
print(out.shape)
&gt; torch.Size([64, 3, 24, 24])
</code></pre>
<p>CC <a class="mention" href="/u/josuecom">@JosueCom</a></p>

Gantavya_Bhatt1: <p>Happy to see this development in the latest version of torchvision. Kudos <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Iterable style dataset exhausts after 1 epoch

Charan0: <p>It was my first time doing an NLP task / implementing a RNN and I was using the torchtext library to load and do sentiment  analysis on the IMDB dataset. I was able to use <code>collate_fn</code> and <code>batch_sampler</code> and create a DataLoader but it gets exhausted after a single epoch. I’m not sure if this is the expected behavior, if it is then do I need to initialize a new DataLoader for every epoch? If not is something wrong with my implementation, please provide me the correct way to implement the same.</p>
<p>PS. I was following the official <a href="https://github.com/pytorch/text/blob/master/examples/legacy_tutorial/migration_tutorial.ipynb" rel="noopener nofollow ugc">changelog</a> of torchtext from github</p>
<p>You can find my implementation <a href="https://colab.research.google.com/drive/1Ep0s2D2d7fNhhUrYgPTtd5A74FPKsytg?usp=sharing" rel="noopener nofollow ugc">here</a></p>
Export fp16 model to ONNX

Hiperdyne19012: <p>Exporting fp16 Pytorch model to ONNX via the exporter fails. How to solve this?</p>

addisonklinke: <p>Most discussion around quantized exports that I’ve found is on this <a href="https://discuss.pytorch.org/t/onnx-export-of-quantized-model/76884/20">thread</a>. However, most users are talking about int8 not fp16 - I’m not sure how similar the approaches/issues are between the two precisions</p>
Size of exported onnx model

KUSHAL_JAIN: <p>I trained a model(model A) using pytorch whose forward method depends on the outputs from another model(model B). Model B can be thought of as a feature extractor for model A.<br>
I saved the <code>state_dict</code> of model A only after training it. As expected, the size of the saved <code>.pth</code> file was small since I only saved the trained parameters of model A.<br>
However, when I try to export this model A into onnx format using <code>torch.onnx.export</code>, the size of the resultant onnx model is very large and takes the parameters of model B too. Is this the expected behavior? I know onnx model runs the forward (which involves calling model B) method of model A before exporting it. But does it necessarily have to include the parameters of model B along with model A?</p>

addisonklinke: <p>For the ONNX file to run standalone, it has to contain both the architecture definition and all model weights required to compute the forward path. Given this, it makes sense to me that model B parameters would need to be included</p>
Loading pretrained models caused dead kernel

chocolocked: <p>I tried to load pre-trained resnet18/50 for feature extraction for MIL. but everytime I used ‘pretrained=True’. it gave me a dead kernel. (it used to work perfectly)</p>
<p>I’ve tried two ways:</p>
<ol>
<li>model = models.resnet18(pretrained = True)<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/0/c01d704a31b7e436ddbd9a95a1fc813c6350e85b.png" data-download-href="https://discuss.pytorch.org/uploads/default/c01d704a31b7e436ddbd9a95a1fc813c6350e85b" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/0/c01d704a31b7e436ddbd9a95a1fc813c6350e85b_2_690x67.png" alt="image" data-base62-sha1="rpwKgNi4O7BgTh40Un0MrMVzxHR" width="690" height="67" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/0/c01d704a31b7e436ddbd9a95a1fc813c6350e85b_2_690x67.png, https://discuss.pytorch.org/uploads/default/original/3X/c/0/c01d704a31b7e436ddbd9a95a1fc813c6350e85b.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/0/c01d704a31b7e436ddbd9a95a1fc813c6350e85b.png 2x" data-dominant-color="E8E3E7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">979×96 9.44 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></li>
<li>state_dict = torch.hub.load_state_dict_from_url(url = ‘<a href="https://download.pytorch.org/models/resnet50-19c8e357.pth" rel="noopener nofollow ugc">https://download.pytorch.org/models/resnet50-19c8e357.pth</a>’, model_dir= ‘data/’, file_name = ‘resnet50.pth’)<br>
model= models.resnet50()<br>
model.load_state_dict(torch.load(‘data/resnet50.pth’))<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/5/f527c1a9831b107c693cb317868d3e0078e059fb.png" data-download-href="https://discuss.pytorch.org/uploads/default/f527c1a9831b107c693cb317868d3e0078e059fb" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/5/f527c1a9831b107c693cb317868d3e0078e059fb.png" alt="image" data-base62-sha1="yYK9NWhWKAhagAtzFYRaN7yqpwD" width="690" height="68" data-dominant-color="DEE1E9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">791×78 7.32 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></li>
</ol>
<p>Also i tried like Vgg18 it threw me the same error also. Any help would be greatly appreciated.</p>

nikhil6041: <p><a class="mention" href="/u/chocolocked">@chocolocked</a> did u try updating ur conda environment? i guess it could be because of older packages or version mismatch in case u used pip for installing several libraries.try creating a fresh pytorch env i guess this might solve the issue.</p>
AssertionError:Torch not compiled with CUDA enabled

Bhavya_Soni: <p>I got this error after updating torch and torchvision :- Torch not compiled with CUDA enabled.<br>
Before this I was able to train model on GPU.<br>
Current version of torch(after updating) is 1.9.0+cpu.<br>
After searching on internet I updated my Cuda 11.0 to 11.3 and GTX1050ti driver version 451.82 to 466.77.<br>
It’s not working still, what should I do now?</p>

eqy: <p>Where are you installing your version of torch from? It could be that the binary package was not compiled with CUDA support.</p>

ptrblck: <aside class="quote no-group" data-username="eqy" data-post="2" data-topic="124287">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/eqy/48/46233_2.png" class="avatar"> eqy:</div>
<blockquote>
<p>It could be that the binary package was not compiled with CUDA support.</p>
</blockquote>
</aside>
<p>It indeed seems to be the case:</p>
<aside class="quote no-group" data-username="Bhavya_Soni" data-post="1" data-topic="124287">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bhavya_soni/48/38101_2.png" class="avatar"> Bhavya_Soni:</div>
<blockquote>
<p>Current version of torch(after updating) is 1.9.0+cpu.</p>
</blockquote>
</aside>
<p><a class="mention" href="/u/bhavya_soni">@Bhavya_Soni</a> make sure you are specifying the desired CUDA runtime version when installing the binaries as given <a href="https://pytorch.org/get-started/locally/">here</a>.</p>

Bhavya_Soni: <p>I was getting some package not found error so I tried to do pip install --upgrade torch torchvision.<br>
After that I got this error.</p>

Bhavya_Soni: <p>Looks like PyTorch doesn’t support CUDA 11.3 right now. It’s showing for CUDA 11.1 + torch 1.9.0<br>
So do I need to downgrade the CUDA version? <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

ptrblck: <p>Not necessarily. The binaries ship with their CUDA runtime (as well as cudnn, NCCL etc.) so your local CUDA toolkit installation won’t be used unless you are building PyTorch from source or are compiling a custom CUDA extension.</p>

Bhavya_Soni: <p>So how can I do that ? I mean “CUDA toolkit installation won’t be used unless you are building PyTorch from source or are compiling a custom CUDA extension” how can I do this?</p>

ptrblck: <p>You can just install the NVIDIA driver, select a desired PyTorch binary using the posted link, and install it.<br>
E.g.</p>
<pre><code class="lang-auto">conda install pytorch torchvision torchaudio cudatoolkit=10.2 -c pytorch
</code></pre>
<p>will install PyTorch with the CUDA 10.2 runtime (as well as cudnn7.6.5).</p>

Bhavya_Soni: <blockquote>
<p>pip install torch==1.9.0+cu102 torchvision==0.10.0+cu102 torchaudio===0.9.0 -f <a href="https://download.pytorch.org/whl/torch_stable.html" rel="noopener nofollow ugc">https://download.pytorch.org/whl/torch_stable.html</a></p>
</blockquote>
<p>This was helpful , Thank you.</p>
Non-uniformly changing convolution weights. Is it possible?

csbalint: <p>Let’s suppose we have convolutional module. Right now it is boring:</p>
<pre><code class="lang-auto">class My_Conv(nn.Module):
    def __init__(self, kwargs):
        super(My_Conv, self).__init__()
        
        self.conv = nn.Conv2d(**kwargs)
    def forward(self, x):
        return self.conv(x)
</code></pre>
<p>But I want to do something like this:</p>
<pre><code class="lang-auto">class My_Conv(nn.Module):
    def __init__(self, kwargs):
        super(My_Conv, self).__init__()
        self.something = Something()
        self.conv = nn.Conv2d(**kwargs)
    def forward(self, x):
        alpha=self.something(x)
        return self.conv(x,weight_multipliers=alpha)
</code></pre>
<p>Let’s suppose I have an alpha tensor. Alpha contains multipliers for the convolution weights. But not globally for the whole convolution. It contains different multipliers for each stride of the convolution window. Think of it as some kind of attention.<br>
Now I want to get the convolution outputs just as before, but first I want the weights to be multiplied by the relevant numbers from alpha, for each given window stride.</p>
<p>Is this possible to do efficiently in Pytorch? Is there a hack for this without reimplementing the convolution operation? I looked in the source, and I don’t feel like messing around on the C++ level.</p>

ptrblck: <p>I think the easiest approach would be to <code>unfold</code> the inputs, apply the weights to each patch, and use a matrix multiplication approach for the convolution, which will most likely use a lot of memory and be slow.</p>
<p>Checking the native conv implementation in C++ could yield a speedup, but I also understand this wouldn’t be interesting to you.</p>

csbalint: <p>Thank you for the suggestion! I will try it, and share my results. If it works well in python (apart from the performance)  maybe it will motivate me to seek a more efficient solution such as hacking away in C++.</p>
Selectively disable LSTM dropout

conv8d: <p>I have a 2-layer LSTM (<code>torch.nn.LSTM</code>) model, with dropout enabled.<br>
For a task, I need the intermediate gradients as well, I do this by using backward hooks.<br>
But when I call <code>.backward()</code>, I get the following error:</p>
<pre><code class="lang-auto">cudnn RNN backward can only be called in training mode
</code></pre>
<p>I can bypass this by setting the LSTM layer to <code>.train()</code> but this also enables the dropout.<br>
Can this be fixed?</p>

ptrblck: <p>You can call <code>.train()</code> on the <code>nn.LSTM</code> module alone or disable cudnn for this layer.</p>

YH-WEI: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/6/36d04c6d23b1aaec0a0c1154f2d3488e271958c1.png" alt="image" data-base62-sha1="7OU3Mmy6QMW30QFtwWkg9S8qTUB" width="340" height="297"><br>
Hello!<br>
I wish to ask you a question that how to make the output of the LSTM network in Pytorch become the input of the next step?</p>
<p>I need to move the value output by LSTM to the next input, and then make predictions step by step. How do I describe the code?</p>

conv8d: <p>For the LSTM dropout, I use the <code>dropout</code> parameter of <code>nn.LSTM</code>. When I call <code>.train()</code> on <code>nn.LSTM</code>, the dropout is enabled as well.</p>

ptrblck: <p>In that case you could still disable cudnn for this layer only or set the <code>dropout</code> to zero after calling <code>.train()</code> on it.</p>
Trying to create parallel RNN technical proof of concept

charlesd: <p>This is a technical proof of concept prior to experimenting with an MoE RNN.</p>
<p>Essentially, I want to run two RNNs in parallel on the same input sequence. The caveat is that the number of parallel RNNs to run is unknown - thus the for loop.</p>
<pre><code class="lang-auto">class pRNN(nn.Module):

    def __init__(self, input_size=0, hidden_size=0, num_layers=1, bidirectional=False, dropout=0, subunit_count=1):

        super().__init__()

        self.subunit_count = subunit_count

        subunit_size = math.ceil(hidden_size / subunit_count)

        self.rnn = []
        hidden_size_remaining = hidden_size
        for i in range(self.subunit_count):
            self.rnn.append(nn.GRU(input_size=input_size, hidden_size=min(hidden_size_remaining, subunit_size), num_layers=num_layers, bidirectional=bidirectional, dropout=dropout))

        self.rnn = nn.ModuleList(self.rnn)

    def forward(self, x):

        out = None
        for i in range(self.subunit_count):
            if out is None:
                out, hidden = self.rnn[i](x)
            else:
                out2 = self.rnn[i](x)
                out = torch.cat((out, out2[0]), dim=-1)
                hidden = torch.cat((hidden, out2[1]), dim=-1)

        return out, hidden
</code></pre>
<p>Unfortunately, this is maybe the fifth failed attempt to come up with a technique that runs in parallel - no matter what I try it runs sequentially (including reducing the already minimal batch size of 128 by the number of RNNs). I am running on a single V100.</p>
<p>Any suggestions are appreciated.</p>

ptrblck: <p>You could check, if your GPU has any resources left for a parallel execution of the RNNs.<br>
RNNs are using matrix multiplications internally, which are executed via cublas on the GPU. Generally matmuls tend to saturate the device as much as possible, so that other kernels might have to wait for their execution. You could check the profile via e.g. Nsight Systems and also take a look at e.g. <a href="https://github.com/pytorch/pytorch/issues/59692#issuecomment-859173905">this post</a>.</p>
Does pytorch have some run time data generator ? like data generator in keras?

nsknsl: <p>Is is possible to generate batch data at the running time using pytorch?<br>
for example , my training data consists of 30000 256*256 images, can I generate batch data containing, for example ,10 images at each iteration,  thus I don’t have to load the whole 30000 images one time but load them step by step?<br>
Thank you.</p>

smth: <p>have you looked at our examples much? most of the vision ones do this: <a href="https://github.com/pytorch/examples/">https://github.com/pytorch/examples/</a></p>

Raj_Roy: <p>Hi ,<br>
Did you find what you were looking for ?</p>

ptrblck: <p>The <a href="https://pytorch.org/tutorials/beginner/basics/data_tutorial.html">Dataset &amp; DataLoader</a> tutorial explains how lazy loading can be used and how the <code>DataLoader</code> creates the batches, so you might want to take a look at it. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Question re: F.max_pool2d parameters

agnestang: <p>I’m going through the Pytorch Recipe: Defining a Neural Network in Pytorch, and I have a question regarding the parameters for F.max_pool2d.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/e/ae8f3540a54b96f12c301fa6e61fec2a3e5ffe25.png" data-download-href="https://discuss.pytorch.org/uploads/default/ae8f3540a54b96f12c301fa6e61fec2a3e5ffe25" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/e/ae8f3540a54b96f12c301fa6e61fec2a3e5ffe25_2_554x500.png" alt="image" data-base62-sha1="oUdTTPvSVywDyOMqMqp5rLAk8Bf" width="554" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/e/ae8f3540a54b96f12c301fa6e61fec2a3e5ffe25_2_554x500.png, https://discuss.pytorch.org/uploads/default/original/3X/a/e/ae8f3540a54b96f12c301fa6e61fec2a3e5ffe25.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/e/ae8f3540a54b96f12c301fa6e61fec2a3e5ffe25.png 2x" data-dominant-color="EDEDF0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">600×541 45.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>In the above code, there are two parameters for F.max_pool2d: x and 2.</p>
<p>My questions:</p>
<ol>
<li>To clarify, is the first parameter ‘x’ just representing our data?</li>
<li>I think I read somewhere that the second parameter is the kernel size, but I’m confused because I thought pooling doesn’t use kernels (aka filters)? I thought only the convolution operation uses kernels/filters? What is this kernel doing in the pooling operation? Or does this second parameter represent something else?</li>
</ol>
<p>Any help would be appreciated, thank you!</p>

ptrblck: <ol>
<li>Yes, <code>x</code> is the input activation.</li>
<li>Pooling layers do use a kernel (which doesn’t contain trainable weights) and you can imagine it as a “window” or “patch” which is selecting the input locations for the pooling operation. E.g. a kernel size of 2 would select a <code>2x2</code> window (= 4 pixels) and in case of <code>nn.MaxPool2d</code> it would then return the max. value of these 4 inputs in the output activation.</li>
</ol>
The parameters of "path" and :text_fields" in using wikitext2 in Torchtxt

Huaiyang_Gongzi: <p>In the tutorial of [pytorch tutorial of language modeling]<br>
(<a href="https://pytorch.org/tutorials/beginner/transformer_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Language Modeling with nn.Transformer and TorchText — PyTorch Tutorials 1.9.0+cu102 documentation</a>)</p>
<p>There is code of using wikitext2```<br>
import torch<br>
from torchtext.datasets import WikiText2</p>
<p>train_iter = WikiText2(split=‘train’)</p>
<pre><code class="lang-auto">It causes the error message shown as follows, which indicates that I need to set up "path" and "text_field".  However, the Pytorch documents does not specify the physical meanings of these two parameters? 

</code></pre>
<hr>
<p>TypeError                                 Traceback (most recent call last)<br>
 in <br>
4 from torchtext.vocab import build_vocab_from_iterator<br>
5<br>
----&gt; 6 train_iter = WikiText2(split=‘train’)<br>
7 tokenizer = get_tokenizer(‘basic_english’)<br>
8 vocab = build_vocab_from_iterator(map(tokenizer, train_iter), specials=[""])</p>
<p>TypeError: <strong>init</strong>() missing 2 required positional arguments: ‘path’ and ‘text_field’</p>
<pre><code class="lang-auto"></code></pre>
Is it possible to find the inverse of AdaptiveAvgPool2d?

Surayuth_Pintawong: <p>For example,</p>
<pre><code class="lang-auto">x = torch.rand(1, 3, 7, 7)  
m = torch.nn.AdaptiveAvgPool2d(output_size=1)
y = m(x)     #   y.shape = (1, 3, 1, 1)
</code></pre>
<p>Is it possible to obtain x via y?</p>

ptrblck: <p>You won’t be able to calculate the input values of the mean calculation.<br>
However, if you are more concerned about the shapes (not the exact values) you could use e.g. <code>nn.ConvTranspose2d</code>.</p>
Can I configure a tokenizer based on particular model configuration?

Shrabani_Ghosh: <p>The purpose here, I have two pretrained models, source model and target model which are trained based on masked language modeling with RoBERTa-base with source data and target data for domain adaptation.<br>
Now, when I tokenize the source data with pretrained model, I want to use that model configured tokenizer.</p>
<p>And, initialize the target model with source model weights so that target model has some information about the source model. And also want to configure the tokenizer based on target model configuration.</p>
<p>pretrained_model = ‘roberta-pretrain-HON-24T_final’<br>
pretrained_path = f’models/{pretrained_model}’<br>
config = AutoConfig.from_pretrained(f’{pretrained_path}/config.json’)<br>
src_model = AutoModel.from_pretrained(pretrained_path, config=config)<br>
src_encoder = AutoTokenizer.from_pretrained(src_model)</p>
<p>pretrained_model = ‘roberta-pretrain-ICWSM-85T_final’<br>
pretrained_path = f’models/{pretrained_model}’<br>
config = AutoConfig.from_pretrained(f’{pretrained_path}/config.json’)<br>
tgt_model = AutoModel.from_pretrained(pretrained_path, config=config)<br>
tgt_encoder = AutoTokenizer.from_pretrained(src_model)</p>
<p>tgt_encoder.load_state_dict(src_encoder.state_dict())</p>
<p>I want to do the same for classifier, I was able to do with classifier, but not able to do for the encoder.<br>
How to do that?</p>
Can I allocate intermediate variable before optimization loop?

Haozhi: <p>If I have several large intermediate variable in the optimization loop, can I allocate them outside the loop to avoid to generate them every iteration? I encountered RuntimeError when trying to do so and I found an <a href="https://discuss.pytorch.org/t/runtimeerror-trying-to-backward-through-the-graph-a-second-time-but-the-buffers-have-already-been-freed-specify-retain-graph-true-when-calling-backward-the-first-time/6795/29">explanation</a>. Is there a correct way to allocate intermediate variable?</p>
<pre><code class="lang-auto">var1 = torch.rand(128, 128, requires_grad=True)
m1 = var1.new_empty((4, *var1.shape)) # intermediate variable
a = torch.linspace(1, 10, 4).reshape(4, 1, 1)

for i in range(10):
    #m1 = var1.new_empty((4, *var1.shape)) # No error if this line is uncommented
    m1[:] = torch.exp(var1[None, :] ** 2)
    l = loss(m1, data)
    l.backward()
    with torch.no_grad():
        var1 -= var1.grad * 0.1
        var1.grad.zero_()

#RuntimeError: Trying to backward through the graph a second time, but the saved intermediate results #have already been freed. Specify retain_graph=True when calling .backward() or autograd.grad() the #first time.
</code></pre>

ptrblck: <p>Instead of recreating <code>m1</code> (this could also work, since PyTorch would reuse the allocated memory) you could also detach <code>m1</code> from the previous iteration using <code>m1.detach_()</code>.</p>

Haozhi: <p>Got it. Thanks a lot.</p>

Haozhi: <p>Could you please show me how to detach <code>m1</code>? I used the following code but an error happened.</p>
<pre><code class="lang-auto">var1 = torch.ones(128, 128, requires_grad=True)
m1 = var1.new_empty((4, *var1.shape), requires_grad=True)
a = torch.linspace(1, 10, 4).reshape(4, 1, 1)

for i in range(10):
    print(i)
    m1.detach()[:] = torch.exp(var1[None, :] * 2)
    l = loss(m1, data)
    l.backward()
    with torch.no_grad():
        var1 -= var1.grad * 0.1
        var1.grad.zero_()
#---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
&lt;ipython-input-31-7971284f49be&gt; in &lt;module&gt;
      9     l.backward()
     10     with torch.no_grad():
---&gt; 11         var1 -= var1.grad * 0.1
     12         var1.grad.zero_()

TypeError: unsupported operand type(s) for *: 'NoneType' and 'float'
</code></pre>

ptrblck: <p>Here is an example:</p>
<pre><code class="lang-python">var1 = torch.rand(128, 128, requires_grad=True)
m1 = var1.new_empty((4, *var1.shape)) # intermediate variable
a = torch.linspace(1, 10, 4).reshape(4, 1, 1)
data = torch.randn(128, 128)

for i in range(10):
    m1.detach_()
    m1[:] = torch.exp(var1[None, :] ** 2)
    l = (m1 - data).mean()
    l.backward()
    with torch.no_grad():
        var1 -= var1.grad * 0.1
        var1.grad.zero_()
</code></pre>
Get 'out of memory error' with on an empty card

MrChenFeng: <p>Hi guys,<br>
I got a very strange problem.<br>
I’m now experimenting with two completely same cards and same code, both of them are free of tasks. But I will get ‘out of memory error’ with one of them. Anyone ever encountered similar problems?</p>

eqy: <p>Can you share the concrete error message you are getting?</p>
Vectorizing index selection by nested dimension mean condition

bmahlbrand: <p>I have the following snippet, I’m wondering how to effectively vectorize this in ‘pure’ pytorch:</p>
<pre><code class="lang-auto">indices = []
for i in range(0, self.dataset.shape[0]):
    if torch.mean(self.dataset[i]) &gt;= .2:
        indices.append(i)

self.dataset = self.dataset[indices, :, :, :, :]
self.target = self.target[indices, :, :, :, :]
</code></pre>
<p>I tried<br>
<code>b = torch.mean(self.dataset, dim=0) &gt; .2</code><br>
<code>indices = b.nonzero()</code></p>
<p>and</p>
<p><code>torch.where(torch.mean(self.dataset[:, ???]) &gt; .2, self.dataset, torch.FloatTensor([0.0]))</code></p>
<p>to no avail. Any thoughts?</p>

bsridatta: <p>The first one should work, it would be nice to see the expected output and the one you get.</p>
<p><code>a = torch.FloatTensor(3,3)</code><br>
<code>b = a.mean(dim=0) &gt; 2</code><br>
<code>c = b.nonzero()</code></p>
<p><code>a - &gt; tensor([[-2.8721e+27,  4.5780e-41, -2.8721e+27],         [ 4.5780e-41,  0.0000e+00,  0.0000e+00],         [ 0.0000e+00,  6.8929e+34,  8.5771e-39]])</code></p>
<p><code>b -&gt; tensor([False,  True, False])</code></p>
<p><code>c -&gt; tensor([[1]])</code></p>

bmahlbrand: <p>given the following:</p>
<blockquote>
<pre><code>dataset = torch.randn((123, 1, 64, 64, 64))
b = torch.mean(dataset, dim=0) &gt; .2
c = b.nonzero()
print(c.shape)
</code></pre>
</blockquote>
<p>this prints:</p>
<blockquote>
<p>torch.Size([3495, 4])</p>
</blockquote>
<p>where I expect it to be more like assuming that 23 of the original 123 3d volume elements don’t exceed the threshold</p>
<blockquote>
<p>torch.Size(100, 1, 64, 64, 64)</p>
</blockquote>
DataLoader: iterate through a subset of data instead of a the whole everytime it is called

AerysS: <p>Let’s say I have a torch <code>dataloader = DataLoader(...)</code> object. I don’t want to iterate through the whole dataset whenever I call <code>for data, label in dataloader:</code> in a function, so currently I use:</p>
<pre><code class="lang-python">dataloader = DataLoader(...)
iter_dataloader = iter(dataloader)
batch = iter_dataloader.next()  # Set the first batch

def train_batch():
   data, label = batch
   prediction = model(data)
   # Do fancy things here
   try: 
      batch = iter_dataloader.next()  # Load the next batch
   except:
      iter_dataloader = iter(dataloader)  # if the iterator object reaches the end, reset the dataloader
      batch = iter_dataloader.next()  

for _ in range(N):
   train_batch()  # This function is called multiple times
</code></pre>
<p>For each call of <code>train_batch()</code>, I get a batch from the dataset, train the model, and load the next batch. If there is no batch left, I reset the DataLoader object.</p>
<p>Now to my questions:</p>
<ol>
<li>Is there a way to have a cleaner code? That is, I don’t want to use the <code>iter</code> and <code>next</code> method. Everytime I call it, it auto samples a batch from it and auto resets when it reaches the end. I hear about the <code>Sampler</code>, but I have not used it.</li>
<li>Extension of above: instead of a batch, can I have like <code>K</code> batches or <code>1/K</code> of the dataset’s size I am using?</li>
<li>I want three sampling methods: (1) sampling batches from it in sequence (no shuffle), (2) sampling from it randomly (with and without replacement - shuffle), and (3) sampling from it such that the labels are equal. Is there a way to do this?</li>
</ol>