akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Cannot import torch after a succesful custom pytorch build

Martin_Kodovsky: <p>Hello, I’m experiencing the following trouble:</p>
<pre><code class="lang-auto">Python 3.11.1 (main, Jan  6 2023, 00:00:00) [GCC 12.2.1 20221121 (Red Hat 12.2.1-4)] on linux
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt; import torch
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/home/user/path/pytorch/torch/__init__.py", line 932, in &lt;module&gt;
    from ._tensor import Tensor
  File "/home/user/path/pytorch/torch/_tensor.py", line 12, in &lt;module&gt;
    import torch.utils.hooks as hooks
  File "/home/user/path/pytorch/torch/utils/__init__.py", line 7, in &lt;module&gt;
    from .backend_registration import rename_privateuse1_backend
  File "/home/user/path/pytorch/torch/utils/backend_registration.py", line 1, in &lt;module&gt;
    from torch._C import _rename_privateuse1_backend
ImportError: cannot import name '_rename_privateuse1_backend' from 'torch._C' (/home/user/path/pytorch/torch/_C.cpython-311-x86_64-linux-gnu.so)
&gt;&gt;&gt; 

</code></pre>
<p>I used this building setup when compiling pytorch:<br>
<code>BUILD_TEST=0 CXX="/home/user/path/gcc_custom/bin/g++" CC="/home/user/path/gcc_custom/bin/gcc" CMAKE_CXX_FLAGS="-march=znver3 -fPIE" python setup.py develop --install-dir ~/.local/lib/python3.11/site-packages/</code></p>
<p>I had the pytorch switched to nightly branch (git 0a73b08b9f2)</p>
<p>I used gcc 11.1.0 and cuda 12.0 toolkit to link up to. The built was successful without any issues (at least as I could see).</p>
<p>I tried to inspect the**_C.cpython-311-x86_64-linux-gnu.so** and I was surprised that basically all of the static functions from torch/csrc/Module.cpp are missing (or perhaps I’m looking at wrong place, I’m not sure).</p>
<p>Has anyone solved this kind of problem. Or does anybody already know what I’m doing wrong from the description of the issue?</p>
<p>Thanks a lot</p>

ptrblck: <p>Are you running your REPL in the source directory?</p>
<pre><code class="lang-auto"> File "/home/user/path/pytorch/torch/utils/backend_registration.py", line 1, in &lt;module&gt;
    from torch._C import _rename_privateuse1_backend
ImportError: cannot import name '_rename_privateuse1_backend' from 'torch._C' (/home/user/path/pytorch/torch/_C.cpython-311-x86_64-linux-gnu.so)
</code></pre>
<p>The error points to source files located in your <code>/home</code> while you were trying to install it into a <code>site-packages</code> folder located in your <code>/home</code><br>
If so, switch the <code>wdir</code> and check if this would fix the error.</p>

Martin_Kodovsky: <p>Thank you a lot! I was staring at it for too long. You were right about <code>wdir</code>; and probably I had also have LD_LIBRARY_PATH crippled too. Thanks once again</p>

serser: <p>In my case, it appears to be that <code>LD_LIBRARY_PATH</code> was wrongly set, two versions of pythons were installed. Removing the unnecessary one will do the work.</p>
Torch.utils.data.random_split using all RAM

ekrx: <p>I have a custom dataset that is lazy loaded with 170 million samples. When I call the method <code>torch.utils.data.random_split</code> to split between train and test, it takes a very long time. I can see that the memory utilization goes all the way to 100% and OS starts swapping, I have 32gb of physical memory. It is my understanding that  <code>torch.utils.data.random_split</code> does not loads the data into memory. So why is <code>random_split</code> taking so much memory for this operation?</p>

ptrblck: <p><code>random_split</code> is creating <code>Subset</code>s as seen <a href="https://github.com/pytorch/pytorch/blob/d04faf4531aa499e0f03bca1c97224e7b2f47b84/torch/utils/data/dataset.py#L486">here</a> which wrap the <code>Dataset</code> and call it with the passed indices as seen <a href="https://github.com/pytorch/pytorch/blob/d04faf4531aa499e0f03bca1c97224e7b2f47b84/torch/utils/data/dataset.py#L392-L422">here</a>. I don’t see where the <code>Dataset</code> is cloned unless the memory usage increase is not caused by <code>random_split</code> but e.g. the workers in the <code>DataLoader</code>.</p>

ekrx: <p>The dataset is not responsible for that. It only contains references to files, and only read line by line once and only when the <strong>getitem</strong> method is called.</p>
<p>If I dont call the <code>random_split</code> method and just iterate over the dataset, I see constant memory utilization without any observable increase.</p>
<p>Can it be that just holding the list of the 170 M indices is causing this memory spike?</p>

ptrblck: <p>I doubt it, as 170,000,000 indices would use ~1.2GB:</p>
<pre data-code-wrap="python"><code class="lang-python">lengths = [100000000, 70000000]
indices = torch.randperm(sum(lengths)).tolist()

print(sys.getsizeof(indices) / 1024**3)
# 1.266598753631115
</code></pre>
<p>If your memory usage is already high, these additional 1.2GB could indeed force your system to offload into the swap.</p>
Why VAE loss doesn't converge to zero?

InfT: <p>Hi everyone!<br>
I’m using a Variational Autoencoder and this is my implementation for the loss function:</p>
<pre><code class="lang-auto">class VariationalAutoencoder(nn.Module):
    # ...some functions...

    def gaussian_likelihood(self, x_hat, logscale, x):
        scale = torch.exp(logscale)
        mean = x_hat
        dist = torch.distributions.Normal(mean, scale)
        # measure prob of seeing image under p(x|z)
        log_pxz = dist.log_prob(x)
        return log_pxz.sum(dim=(1, 2, 3))

    def forward(self, input):
        mu, logvar = self.encode(input)
        z = self.reparameterise(mu, logvar)
        return self.decoder(z), mu, logvar, z

    def loss_function(self, x_hat, x, mu, logvar, β=1):
        std = torch.exp(logvar / 2)
        q = torch.distributions.Normal(mu, std)
        z = q.rsample()

        # reconstruction loss
        recon_loss = self.gaussian_likelihood(x_hat, self.log_scale, x)

        # kl
        kl = self.kl_divergence(z, mu, std)

        # elbo
        elbo = (kl - recon_loss)
        elbo = elbo.mean()
        return elbo

    def kl_divergence(self, z, mu, std):
        # --------------------------
        # Monte carlo KL divergence
        # --------------------------
        # 1. define the first two probabilities (in this case Normal for both)
        p = torch.distributions.Normal(torch.zeros_like(mu), torch.ones_like(std))
        q = torch.distributions.Normal(mu, std)

        # 2. get the probabilities from the equation
        log_qzx = q.log_prob(z)
        log_pz = p.log_prob(z)

        # kl
        kl = (log_qzx - log_pz)
        kl = kl.sum(-1)

        return kl
</code></pre>
<p>I use Sigmoid() function when exiting the decoder. I train the model in this way:</p>
<pre><code class="lang-auto">for epoch in range(0, epochs + 1):
        if epoch &gt; 0:  # test untrained net first
            model.train()
            train_loss = 0
            loop = tqdm(train_loader)
            optimizer = model.setOptimizer(model)
            for x in loop:
                x = x.to(device)
                x_hat, mu, logvar, features = model(x)
                loss = model.loss_function(x_hat, x, mu, logvar)
                train_loss += loss.item()
                optimizer.zero_grad()
                loss.backward()
                optimizer.step()
                loop.set_postfix(loss=loss)
            train_loss = train_loss /= len(train_loader.dataset)
            print(f'====&gt; Epoch: {epoch} Average loss: {train_loss:.4f}')
</code></pre>
<p>The loss doesn’t settle at zero but but becomes negative (around -2). If I remove <code>train_loss = train_loss /= len(train_loader.dataset)</code>, it’s diverges completely.</p>
<p>How can I make the loss converge to zero?</p>

banikr: <p>Hello, have you figured this out?<br>
I am dealing with the same problem</p>
I have a problem Radeon RX 7900 XTX

Levan_Gelashvili: <p>Hello, I have a problem. Please, help. I’m using amd 7900 xtx.<br>
When I start training after 1-2 hours the training process stops randomly. Without any error. Afterwards I tried the following:<br>
When I run torch.cuda.is_available() I get the result true.<br>
When I run print(torch.cuda.get_device_name()) I get a Radeon RX 7900 XTX.<br>
When I run torch.randn(1,2) nothing happens. hangs without response.</p>
<p>After rebooting the computer everything works. But the problem keeps recurring.</p>
<p>Then I tried using nvidia rtx 2060 super with the same code. I don’t see any problems.</p>
<p>For amd I use rocm5.7 and PyTorch Stable (2.2.1)</p>
<p>I can send you my code</p>
Moving tensors to devices

adrianj: <p>I’m trying to do some manual model pipelining simply by using .to(device) approach. I have a network that is built from some nested nn.Module classes. My initial idea was to do the movement of individual layers in the class constructors, and then everything should automatically be handled (except for input/output data movement), i.e.:</p>
<pre><code class="lang-auto">class Encoder(nn.Module):
    def __init__(self, device="cpu", chs=(3,16,32,64,128)):
        super().__init__()
        self.enc_blocks = nn.ModuleList([Block(chs[i], chs[i+1]).to(device) for i in range(len(chs)-1)])
        self.pool       = nn.MaxPool2d(2).to(device)
        self.device     = device
</code></pre>
<p>(Block in the above is another class Block(nn.Module))</p>
<p>However, on experimenting with this, it didn’t actually seem to move those layers on to the device. I had to move the functionality into the forward function instead, i.e.:</p>
<pre><code>def forward(self, x):
    return_values = []
    for block in self.enc_blocks:
        x = block.to(self.device)(x)
        return_values.append(x.to(self.device))
        x = self.pool.to(self.device)(x)
    return return_values
</code></pre>
<p>Which now puts things where I expect them. Is there a reason the tensor movement functions in the constructor wouldn’t work, but the ones in the forward function would work? Or is my thinking about what to() is doing incorrect?</p>
<p>thanks</p>

ptrblck: <aside class="quote no-group" data-username="adrianj" data-post="1" data-topic="198949">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/dfb087/48.png" class="avatar"> adrianj:</div>
<blockquote>
<p>However, on experimenting with this, it didn’t actually seem to move those layers on to the device.</p>
</blockquote>
</aside>
<p>It works for me using:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
import torch.nn as nn

class Encoder(nn.Module):
    def __init__(self, device="cpu"):
        super().__init__()
        self.enc_blocks = nn.ModuleList([nn.Linear(1, 1).to(device) for i in range(3)])
        self.device     = device

    def forward(self, x):
        for module in self.enc_blocks:
            x = module(x)
        return x


model = Encoder(device="cuda:5")
print(model.enc_blocks[0].weight.device)
x = torch.randn(1, 1, device="cuda:5")

out = model(x)
print(out.device)
</code></pre>
<p>Output:</p>
<pre data-code-wrap="python"><code class="lang-python">cuda:5
cuda:5
</code></pre>

adrianj: <p><img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=12" title=":thinking:" class="emoji" alt=":thinking:" loading="lazy" width="20" height="20">Ok, looks like I just needed to upgrade Torch to get the correct behaviour.</p>
<p>Thanks (I’m jealous of your cuda:5 <img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=12" title=":grin:" class="emoji" alt=":grin:" loading="lazy" width="20" height="20"> )</p>
Is SGD doc formula for a single sample or a mini-batch?

v_kubicki_github: <p>Is the formula presented at <a href="https://pytorch.org/docs/stable/generated/torch.optim.SGD.html" class="inline-onebox" rel="noopener nofollow ugc">SGD — PyTorch 2.2 documentation</a> for each sample or for a mini-batch ? I ask because I want to know if for a mini-batch of size n, the formula used internally would be g_t​ ← g_t ​+ n λ_{θ_t−1} instead of g_t​ ← g_t ​+ λ_{θ_t−1}.</p>
<p>Thanks</p>

KFrank: <p>Hi Vincent!</p>
<aside class="quote no-group quote-modified" data-username="v_kubicki_github" data-post="1" data-topic="198951" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/v_kubicki_github/48/67554_2.png" class="avatar"> v_kubicki_github:</div>
<blockquote>
<p>Is the formula … for each sample or for a mini-batch ?</p>
</blockquote>
</aside>
<p>It’s neither.  <code>SGD</code> is unaware of your batch size or even whether you used<br>
a batch at all.</p>
<p>All <code>SGD</code> does is update your parameters with a gradient-descent step based<br>
on the <code>.grad</code> properties of those parameters.  It knows nothing about how<br>
those <code>.grad</code>s were set.  They could come from backpropagating a single<br>
sample, from a batch of samples, from accumulating gradients across multiple<br>
backpropagations by not calling <code>.zero_grad()</code> in between, or from setting<br>
<code>.grad</code> manually according to some scheme.</p>
<p>Best.</p>
<p>K. Frank</p>
nn.Parameters vs nn.Module.register_parameter

dragen: <p>According to the document, nn.Parameter will:</p>
<blockquote>
<p>they are automatically added to the list of its parameters, and will appear e.g. in parameters() iterator</p>
</blockquote>
<p>and nn.Module.register_parameter will</p>
<blockquote>
<p>Adds a parameter to the module.</p>
</blockquote>
<p>I wonder since nn.Parameter will add tensor into parameters automatically, why we need register_parameter function?</p>

4400f61bb0fb95f92bda: <p>I guess it is because there are some tasks to add parameters to given module. Here is an example code. (Below code is just example, but I did this kind of thing when I implemented spectral normalization layer)</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

class Test(nn.Module):
    def __init__(self, module):
        super(Test, self).__init__()
        self.module = module
        self.register_param()
    
    def register_param():
        exist_w = hasattr(self.module, 'w')
        if not exist_w:
            w = nn.Parameter(torch.ones(1))
            self.module.register_parameter(w) # register 'w' to module

    def forward(self, x)
        return x

conv = nn.Conv2d(3, 3)
conv_w = Test(conv)
</code></pre>

blackbirdbarber: <p><code>nn.Module.register_parameter</code> takes the tensor or <code>None</code> but first checks if the name is in dictionary of the module. While <code>nn.Parameter</code> doesn’t have such check.</p>

Max_Unhold: <p>Hey, thank you for that answer. In this example, what type does the self.module have? Is it an nn.Module?</p>
BERT Non Contextual Embeddings? [SOLVED] THANKS FOR THE HELP GUYS!

Jack_Archer: <p>I have been training an image captioning model and realized that I was embedding with Bert contextual embeddings. This completely screwed up training and testing. Is there anyway to get non contextual BERT embeddings?</p>
<p>I’ve tried this i.e. get jus  first layer of BERT but I still think it captures some contextual information. My testing results are much worse than my training after 6 epochs (50,000 samples in dataset):</p>
<p>outputs = self.model(**input_ids, output_hidden_states=True)<br>
# embedding comes from last hidden state of Bert embedding model used here -<br>
emb_caption = outputs.hidden_states[0]</p>
<p>Solution: if you are using BERT embedding there is always going to be some form of contextual information encapsulated in the embedding. The way I solved it was embedding all words generated at each time step during testing. This gave me great captions. However, I was using a CNN-LSTM model. And even though it works, it’s more of a testament to the power of BERT than LSTM architecture so it’s kind of pointless in a way! BERT did the legwork during training leaving the LSTM to do little to no real training.</p>
_cmake_cuda_whole_flag?

RedHead: <pre><code class="lang-auto">CMake Error: Error required internal CMake variable not set, cmake may not be built correctly.
Missing variable is:
_CMAKE_CUDA_WHOLE_FLAG
</code></pre>
<p>when attempting to build from source using python3 setup.py bdist_wheel</p>

ptrblck: <p>Might be related to <a href="https://discourse.cmake.org/t/cmake-cuda-clang-fails/8657/8">this issue</a>.</p>

RedHead: <p>Problem solved, need to launch python setup.py build —cmake-only first</p>
Performance discrepancy in masking operation on GPU-initialized vs. CPU-to-GPU moved tensors

malfonsoarquimea: <p>Hello, PyTorch Community,</p>
<p>I’ve encountered a perplexing performance issue when applying a boolean mask to a tensor, and I’m hoping to get some insights or suggestions on how to address it. The core of the problem lies in the significant time difference in executing a masking operation, depending on how the tensor involved is initialized and where it resides (CPU vs. GPU).</p>
<p>Here’s a simplified version of the scenario:</p>
<pre><code class="lang-auto">
import torch

# Assume num_rays, original_num_light_directions are predefined integers
# visibility is a tensor already on GPU
# Assume mask is predefined and compatible with total_vis dimensions

# Scenario 1: Initializing tensor on CPU, then moving to GPU
total_vis_cpu_to_gpu = torch.ones(num_rays, original_num_light_directions, 1).type_as(visibility)
total_vis_cpu_to_gpu[mask] = visibility

# Scenario 2: Directly initializing tensor on GPU
total_vis_gpu = torch.ones(num_rays, original_num_light_directions, 1, device=visibility.device, dtype=visibility.dtype)
total_vis_gpu[mask] = visibility
</code></pre>
<p>In both cases, <code>total_vis[mask] = visibility</code> is the operation of interest. However, the performance drastically differs:</p>
<ul>
<li>In Case 1, where <code>total_vis</code> is initialized on the CPU and then moved to the GPU, the operation executes relatively fast.</li>
<li>In Case 2, when <code>total_vis</code> is initialized directly on the GPU, the same operation takes significantly longer.</li>
</ul>
<p>This discrepancy is puzzling because the operation that applies the mask to modify <code>total_vis</code> is the same in both scenarios, yet the performance impact is quite different. I initially thought the delay in Case 2 could be due to how memory is allocated or managed on the GPU when tensors are directly initialized there, but I’m not entirely sure.</p>
<p>Has anyone experienced a similar issue or can provide insight into why this might be happening? Any suggestions on how to mitigate this performance difference would be greatly appreciated.</p>
<p>Thank you in advance for your time and help.</p>
Why these tensor have all same values?

citystrawman: <p>I am trying to debug and watch the values of a code for studying machine learning. Here’s the code snippet:</p>
<pre><code class="lang-auto">train_data = DataGenerator(lines[:num_train], input_shape, True)  # num_train is 1066
......
gen_train = DataLoader(train_data, batch_size=4) 
......
epochs = 20 
for epoch in range(epochs):
    total_train = 0  
    for data in gen_train:
        img, label = data    # breakpoint here
        with torch.no_grad():
            img = img.to(device)
            label = label.to(device)
        optim.zero_grad()
        output = net(img)
        train_loss = nn.CrossEntropyLoss()(output, label).to(device)
        train_loss.backward()  
        optim.step()  
        total_train += train_loss  
......
</code></pre>
<p>here in this code, img are 4 images loaded, and label are 4 integers that corresponds to the images’ class.  when I put a breakpoint, I saw the images has 4 “blocks”,  each block has all same values:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/2/6284b4c8b7f2905a1fc049ebedb7374f98a40fc2.png" data-download-href="https://discuss.pytorch.org/uploads/default/6284b4c8b7f2905a1fc049ebedb7374f98a40fc2" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/2/6284b4c8b7f2905a1fc049ebedb7374f98a40fc2.png" alt="image" data-base62-sha1="e3x73rHUvPEFxzNPbrFj3V44dlU" width="549" height="500" data-dominant-color="505355"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">593×540 20.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>(as I am new user, I can only upload one image, but all other 3 blocks has the same value)</p>
<p>these blocks should be the images, but why they all have the same numbers?</p>

ptrblck: <p>Check if these values represent borders with a static color by visualizing the images via e.g. <code>matplotlib</code>.</p>
How to properly install a project that depends on LibTorch?

NiklasKappel: <p>I followed the <a href="https://pytorch.org/cppdocs/installing.html" rel="noopener nofollow ugc">official tutorial</a> to build the example LibTorch app. It works just fine from the build directory, but for a real project, I would like my users to install the app after building it. So I added installation instructions to my <code>CMakeLists.txt</code>:</p>
<pre data-code-wrap="cmake"><code class="lang-cmake"># Code from the official tutorial:
cmake_minimum_required(VERSION 3.18 FATAL_ERROR)
project(example-app)

find_package(Torch REQUIRED)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${TORCH_CXX_FLAGS}")

add_executable(example-app example-app.cpp)
target_link_libraries(example-app "${TORCH_LIBRARIES}")
set_property(TARGET example-app PROPERTY CXX_STANDARD 17)

# New: Install the app:
set(CMAKE_INSTALL_PREFIX "$ENV{HOME}/.local" CACHE PATH "Default install path" FORCE)
include(GNUInstallDirs)
install(
  TARGETS example-app
  LIBRARY
  DESTINATION "${CMAKE_INSTALL_BINDIR}"
)
</code></pre>
<p>With this, <code>make install</code> installs the example app to <code>~/.local/bin/example-app</code>, but from there, it does not work anymore:</p>
<pre data-code-wrap="bash"><code class="lang-bash">$ example-app 
example-app: error while loading shared libraries: libtorch.so: cannot open shared object file: No such file or directory
</code></pre>
<p>Using <code>ldd</code> shows that the example in in the build directory finds the shared libraries that I originally unzipped, while the installed example app does not find them anymore. Lacking experience in C++ deployment, I am not sure how to proceed. Here are my thoughts:</p>
<ol>
<li>Can I tell CMake to also install the shared LibTorch libraries? If so, how? Would that be a good practice considering users may have installed LibTorch already via some other means?</li>
<li>Should I require my users to have LibTorch libraries already installed? If so, by what means should they be installed preferably? Could this cause mismatches between library versions, and should I adjust the <code>CMakeLists.txt</code> to link against already present LibTorch instead of the files from the zip archive?</li>
<li>Can and should I statically link to LibTorch so that my app is self-contained?</li>
</ol>
<p>I realize these question may be more C++ than PyTorch related, but any clarifying input would be very appreciated. <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=12" title=":sweat_smile:" class="emoji" alt=":sweat_smile:" loading="lazy" width="20" height="20"></p>

NiklasKappel: <p>For now, I added the following to my CMake script:</p>
<pre><code class="lang-auto">install(
  DIRECTORY ${torch_SOURCE_DIR}/lib/
  DESTINATION "${CMAKE_INSTALL_LIBDIR}"
)
</code></pre>
<p>Where <code>torch_SOURCE_DIR</code> holds the path to the extracted LibTorch archive. This works, because it simply installs all the shared libraries that LibTorch provides. I am not sure if this is a good practice though.</p>
How can I develop a transformation that performs JPEG compression with a random QF?

eadomenech: <p>I need a transform that performs JPEG compression to the image in question. The QF must be random and belong to a given subset.</p>

Oli: <p>The <a href="https://imgaug.readthedocs.io/en/latest/source/api_augmenters_arithmetic.html#imgaug.augmenters.arithmetic.JpegCompression" rel="nofollow noopener">imgaug</a> library has a jpegcompression that takes a parameter on how much one wants to compress it. I’m guessing QF is quality factor?</p>

eadomenech: <p>And how do I add it to my simple_transform?</p>
<pre><code class="lang-auto"># Transforms
    simple_transform = transforms.Compose(
        [
            transforms.ToTensor(),
            transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
        ]
    )

    # Dataset
    train_dataset = datasets.ImageFolder('data/train/', simple_transform)
    valid_dataset = datasets.ImageFolder('data/valid/', simple_transform)
</code></pre>

Oli: <p>I’ve never done it but you could probably create a <a href="https://pytorch.org/docs/stable/torchvision/transforms.html#torchvision.transforms.Lambda" rel="nofollow noopener">custom transform</a>. Perhaps searching on google for pytorch lambda transform or whatever will help you find some working code of it</p>
<p>Edit: Did just that</p>
<pre><code class="lang-auto">def foo(x):
    return x / 255.0
</code></pre>
<pre><code class="lang-auto">transforms.Lambda(lambda x: foo(x))
</code></pre>

eadomenech: <p>Perfect <a class="mention" href="/u/oli">@Oli</a></p>
<pre><code class="lang-auto">def randomJPEGcompression(image):
    qf = random.randrange(10, 100)
    outputIoStream = BytesIO()
    image.save(outputIoStream, "JPEG", quality=qf, optimice=True)
    outputIoStream.seek(0)
    return Image.open(outputIoStream)

# Transforms
    simple_transform = transforms.Compose(
        [
            transforms.Lambda(randomJPEGcompression),
            transforms.ToTensor(),
            transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
        ]
    )
</code></pre>

Ushavilash: <p>Hi Ernesto,</p>
<p>You can develop a script to randomly select a Quantization Factor from a predefined subset and apply it during JPEG compression. Experiment with different QF values to find the right balance between image quality and file size on the other hand for jpeg compression you can use an online application such as <a href="https://jpegcompressor.com/" rel="noopener nofollow ugc">https://jpegcompressor.com/</a> it compresses images efficiently.</p>
How to do model selection during training?

insomnia_akari: <p>I am currently working on re-implementing the paper “Towards Fast, Accurate and Stable 3D Dense Face Alignment,” and I’m having trouble figuring out the implementation of the Meta-Joint Optimization part. Generally, it seems to be about model selection every k steps. There are two loss functions involved: VDC and wPDC. Every K iterations, the model parameters Theta are updated using these two loss functions, resulting in two different sets of parameters: Theta_VDC and Theta_wPDC. Then, these parameters are validated on a batch, and the one with the better metric is chosen as the initial parameters for the next iteration. This is my implement, but it always give inf loss after several iter.</p>
<pre><code class="lang-auto">model_vdc = mobilenet()
model_wpdc = copy.deepcopy(model_vdc)

optimizer_vdc = torch.optim.AdamW(params=model_vdc.parameters(),lr=lr)
optimizer_wpdc = torch.optim.AdamW(params=model_wpdc.parameters(),lr=lr)

for epoch in range(N):
    for batch_idx,batch in enumerate(trainloader):
        if batch_idx == 0 or (batch_idx != 0 and batch_idx % meta_joint_k != 0):
            # update by vdc loss
            loss_vdc.backward()
            optimizer_vdc.step()
            optimizer_vdc.zero_grad()
           
            # update by wpdc loss
            loss_wpdc.backward()
            optimizer_wpdc.step()
            optimizer_wpdc.zero_grad()
        elif batch_idx != 0 and batch_idx % meta_joint_k == 0:
            model_vdc.eval();model_wpdc.eval()
            # calculate the metric
            ......

            if metric_vdc &gt; metric_wpdc:
                model_vdc.load_state_dict(copy.deepcopy(model_wpdc))
                optimizer_vdc.load_state_dict(copy.deepcopy(optimizer_wpdc))
            else:
                model_wpdc.load_state_dict(copy.deepcopy(model_vdc))
                optimizer_wpdc.load_state_dict(copy.deepcopy(optimizer_vdc))
           model_vdc.training();model_wpdc.training()
</code></pre>
<p>Thank you in advance.</p>
Weights not changing after loading from checkpoint

tooflybryguy: <p>I have defined a function to check whether my model’s weights are changing or not. All is well until I try to resume training from a checkpoint, where both the model and the optimizer state_dicts are saved/loaded. After loading from a checkpoint, my model’s weights show that they are no longer changing. I can re-instanciate my optimizer (defeating the purpose of saving its previous state), and the model’s weights will begin changing again. I have double-checked that the optimizer’s state matches before and after loading (all keys and values within optimizer.<strong>dict</strong>[‘state’] are the same). Does anyone have any suggestions to solve this issue?</p>
Best way to calculate Frobenius norm of high dimensional Jacobian

sofuncheung: <p>Hi friends, I’m trying to figure out the way to calculate the norm of Jacobian between layers of a deep CNN, e.g. the jacobian between the 15th and 18th layer output of VGG19. The problem is these Jacobians are very high dimensional, normal methods like autograd.jacobian will most certainly throw an OOM error. Since I don’t need the exact full Jacobian but it’s Frobenius norm, what is the best way to make it work?</p>
<p>Any help would be much appreciated!</p>
Understanding encoder of Seq2Seq model

helixjolt: <p>I am currently trying to implement a Seq2Seq model for machine translation task. I was following the an official Pytorch tutorial.<br>
<a href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#training" rel="noopener nofollow ugc">Tutorial Link</a></p>
<p>My theoretical understanding for the encoder was that it takes a single input at each timestamps and generates a new hidden state and the process continues i.e. taking the new word and previous hidden state. But in the tutorial they are passing all the input together to the encoder. How is the encoder able to understand the relationship between words if we do this?</p>
<p>I have attached the code snippet below</p>
<pre><code class="lang-auto">class EncoderRNN(nn.Module):
    def __init__(self, input_size, hidden_size, dropout_p=0.1):
        super(EncoderRNN, self).__init__()
        self.hidden_size = hidden_size

        self.embedding = nn.Embedding(input_size, hidden_size)
        self.gru = nn.GRU(hidden_size, hidden_size, batch_first=True)
        self.dropout = nn.Dropout(dropout_p)

    def forward(self, input):
        embedded = self.dropout(self.embedding(input))
        output, hidden = self.gru(embedded)
        return output, hidden


def train_epoch(dataloader, encoder, decoder, encoder_optimizer,
          decoder_optimizer, criterion):

    total_loss = 0
    for data in dataloader:
        input_tensor, target_tensor = data

        encoder_optimizer.zero_grad()
        decoder_optimizer.zero_grad()

        encoder_outputs, encoder_hidden = encoder(input_tensor)      # This line here

        decoder_outputs, _, _ = decoder(encoder_outputs, encoder_hidden, target_tensor)

        loss = criterion(
            decoder_outputs.view(-1, decoder_outputs.size(-1)),
            target_tensor.view(-1)
        )
        loss.backward()

        encoder_optimizer.step()
        decoder_optimizer.step()

        total_loss += loss.item()

    return total_loss / len(dataloader)
</code></pre>

vdw: <p>The <code>nn.GRU</code> class does this all under the hood – that is, it takes the whole input sequence and iterates over each sequence item. In other words, the loop you are looking for is implemented by <code>nn.GRU</code>.</p>
<p>In case you want to implement this loop yourself, because you need to do some customization, you can have a look at <code>nn.GRUCell</code>.</p>
Backpropagation with model ensembling after switching device

OriKatz: <p>I am using ensemble of MLPs.<br>
Following <a href="https://pytorch.org/tutorials/intermediate/ensembling.html" rel="noopener nofollow ugc">this</a> tutorial I tried using vmap to vectorize the computations but I faced difficulties when changing device and backpropagating.<br>
I checked <a href="https://github.com/pytorch/pytorch/issues/120581" rel="noopener nofollow ugc">this</a> issue, however, it does not relate to device switching.</p>
<p>As far as I understand, switching device requires moving the stacked tensors to the new device, how can I do it without detaching them?</p>
<p>Thanks in advance,<br>
Ori.</p>

ptrblck: <p>Could you describe you use case in more details and which issues you are seeing?<br>
A minimal and executable code snippet reproducing errors would be great for debugging it further.</p>
Why images looks different after doing normalization

Mu7annad0: <p>A.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])<br>
ToTensorV2(p=1.0)</p>
<p>is this how the images look like after normalization<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/8/38e4fcd4a997668c8b5153a1f4f020aba562ff98.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/38e4fcd4a997668c8b5153a1f4f020aba562ff98" title="Untitled-1-01"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/8/38e4fcd4a997668c8b5153a1f4f020aba562ff98_2_198x500.jpeg" alt="Untitled-1-01" data-base62-sha1="87jlcS4jItLqMvDnzs9nGPpw9pS" width="198" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/8/38e4fcd4a997668c8b5153a1f4f020aba562ff98_2_198x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/3/8/38e4fcd4a997668c8b5153a1f4f020aba562ff98_2_297x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/8/38e4fcd4a997668c8b5153a1f4f020aba562ff98_2_396x1000.jpeg 2x" data-dominant-color="B2ADAA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Untitled-1-01</span><span class="informations">1920×4832 1.22 MB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>The normalization creates a zero-mean and unit variance output and is thus expected. <code>matplotlib</code> will also <code>clip</code> the values as is mentioned in the usual warning you would get:</p>
<pre data-code-wrap="python"><code class="lang-python">Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).
</code></pre>
Why average pooling is called in vgg16 forward method?

citystrawman: <p>I am a new learner and I am now learning vgg16.</p>
<p>From the following image, vgg only calls maxpooling before FC-4096:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/8/58360825635e456ea0d4471ee73ef14158a12682.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/58360825635e456ea0d4471ee73ef14158a12682" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/8/58360825635e456ea0d4471ee73ef14158a12682_2_496x500.jpeg" alt="image" data-base62-sha1="cAlMmtCwVNmC7GjRWq6KBFSiLAu" width="496" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/8/58360825635e456ea0d4471ee73ef14158a12682_2_496x500.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/5/8/58360825635e456ea0d4471ee73ef14158a12682.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/8/58360825635e456ea0d4471ee73ef14158a12682.jpeg 2x" data-dominant-color="DADADA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">640×644 118 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
However, from its source code, a average pooling is called:</p>
<blockquote>
<pre><code>def forward(self, x):
    x = self.features(x)
    x = self.avgpool(x)
    x = torch.flatten(x, 1)  
    x = self.classifier(x)
    return x
</code></pre>
</blockquote>
<p>so which step corresponds to this  self.avgpool(x) ?</p>
<p>I appreciate that you solve my question. Thank you!</p>

ptrblck: <p>The <code>nn.AdaptiveAvgPool2d((7, 7))</code> is used as a convenient addition to allow the usage of variable input shapes since this layer will output the pre-defined activation shape of <code>[batch_size, nb_channels, 7, 7]</code>, which is the default activation shape when images in the shape <code>[batch_size, 3, 224, 224]</code> are used.</p>

citystrawman: <p>So from my perspective, the original vgg(Simonyan &amp; Zisserman, 2014) does not have this AdaptiveAvgPool2d operation, and it is later introduced for Reduce Overfitting as well as Flexibility?</p>

ptrblck: <p>No, it’s unrelated to overfitting and I’m unsure where this impression comes from. If the original shapes are used, the layer is a no-op since the activation shape has the desired output shape already. However, if the input shapes (and thus the intermediate activation shapes) change, you would receive a <code>RuntimeError</code> previously while the adaptive pooling layer allows you to execute the model.</p>
<p>You can certainly remove the layer and would receive errors for invalid shapes, if you want to stick to the original implementation.</p>