akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Can we train a model with weights initialized with int16?

Sasank_Kottapalli: <p>I am trying to train model with audio signal.<br>
Loaded audio signal with librosa.load and mutiplied with 32768 to convert it into integer and changed the dtype to int16 from float32<br>
Initialized a weight in Channel normalization function(didn’t used nn.layernorm) with int16.<br>
when I tried to train, I recieved the below error<br>
RuntimeError: Only Tensors of floating point dtype can require gradients<br>
cant we use autograd with int16?<br>
will changing all the parameters to int16 solve this issue? if so how to change weights to int16?</p>

KFrank: <p>Hello Sasank!</p>
<p>[quote=“Sasank_Kottapalli, post:1, topic:78381, full:true”]<br>
RuntimeError: Only Tensors of floating point dtype can require gradients<br>
cant we use autograd with int16?<br>
[/quote[</p>
<p>No.  <code>int16</code>s (as well as other integer types) are discrete, and<br>
so are not usefully differentiable.  So you cant use them with<br>
autograd.</p>
<aside class="quote no-group">
<blockquote>
<p>will changing all the parameters to int16 solve this issue?</p>
</blockquote>
</aside>
<p>No, you’ll still have the same problem.</p>
<p>If your network parameters are <code>int16</code>s, and, as part of training,<br>
your optimizer wants to change one of your network parameters<br>
by a small amount, it won’t be able to – it could change the<br>
parameter by <code>1</code> or <code>0</code> (or <code>2</code>, etc.), but not by, say <code>0.0345</code>.</p>
<p>Instead, you should change your ints to floats, something like:</p>
<pre data-code-wrap="python"><code class="lang-python">my_float_tensor = my_int_tensor.float()
</code></pre>
<p>Note, in general, pytorch only likes to perform tensor operation<br>
with tensors of the same type.  So even if you have a case<br>
where it logically and mathematically makes sense to combine<br>
an integer tensor with a float tensor (because, say, gradients<br>
won’t be flowing back through the integer tensor), pytorch won’t<br>
let you (and won’t automatically cast the ints to floats), so you<br>
will have to explicitly cast your int tensor to floats before<br>
performing the operation.</p>
<p>Best.</p>
<p>K. Frank</p>

Sasank_Kottapalli: <p>Thankyou so much Frank for quick reply. Everything made sense <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

christophezei: <p>Hi, for me, I’m just interested in saving my weights as int to be able to make inferences, I don’t need it to do training, is that possible?</p>
Problem in plotting a grid of images

John4: <p>Hello. I’d like to plot a grid of images with the following code:</p>
<pre><code>CHSC = torch.load('calibratedHsCube.pt')
masks = torch.load('mask.pt')
rgb_CHSC = CHSC[:, :, :, (66, 23, 8)]
rgb_CHSC = rgb_CHSC.permute(0, 3, 1, 2)
fig = plt.figure(figsize=(15, 15))
grid_img = torchvision.utils.make_grid(rgb_CHSC, nrow=19)
grid_img = grid_img.permute(1, 2, 0)
plt.imshow(grid_img.detach().numpy())
</code></pre>
<p>The code seems to work but I cannot see any figure. Any suggestion?<br>
Also I used this:</p>
<pre><code>import matplotlib
from matplotlib.figure import Figure
from matplotlib import pyplot as plt
matplotlib.use('TkAgg')
</code></pre>
Efficient on-GPU random tensor perturbation

andrei.chiffa: <p>Hi,</p>
<p>I’ve been using PyTorch for neuroevolution and while it has been successful, I have ran into some performance bottlenecks.</p>
<p>Specifically, I need to derive a number of perturbations for a base tensor stack through a pseudo-random number generator and evaluate if that perturbation improves the loss or not. The critical part is for me to be able to retrieve the random seed used for the perturbation and be able to re-derive the tensor perturbation from it.</p>
<p>So far, to do it, I have performed a deepcopy of the original tensor, followed by a random perturbation, generation of a new perturbed tensor and a forward-propagation pass.</p>
<p>The approximate code of that procedure would be:</p>
<pre><code class="lang-auto">base_state_dict = deepcopy(active_net.main.state_dict()) 
# active_net is the nn.Module containing the ANN model

fixed_rand_seed = secrets.randbelow(&lt; operations to conform to pytorch seed sizing&gt;)
torch.manual_seed(fixed_rand_seed)

new_state_dict = OrderedDict()

for _name, _tensor in base_state_dict.items():
    perturbation_tensor = torch.normal(0.0, tensor_av*diameter, size=_tensor.shape).cuda()
    new_state_dict[_name] = _tensor + perturbation_tensor

active_net.main.load_state_dict(new_state_dict)

loss_fn = nn.CrossEntropyLoss(ignore_index=-1)

for img, lbl in dataloader:
        img = img.cuda()
        lbl = lbl.cuda()

        predict = active_net.forward(img)
        loss = loss_fn(predict, lbl)
        valid_loss += loss.item() * img.size(0)

        valid_loss = valid_loss / len(test_ds_loader.sampler)
</code></pre>
<p>However, this causes a lot of transitions between the CPU and GPU, that does not seem to be easy to remove, given that setting a fixed random seed in pytorch requires a cryptographically secure random seed, that is called using CPU-executed python. Unfortunately, this transfer rapidly becomes a bottleneck in my computational experiments.</p>
<p>Is there a way to achieve the same result more efficiently, without that GPU-CPU-GPU transfer and data re-loading on the GPU?</p>
RuntimeError: cuda runtime error (801) : operation not supported at ..\torch/csrc/generic/StorageSharing.cpp:247

granth_jain: <p>Hi,</p>
<p>I am trying to modify my A3C code from cpu device version to cuda version.</p>
<p>However I am facing above error.</p>
<p>I have already set the start method to ''spawn"</p>
<blockquote>
<p>from torch.multiprocessing import Process, set_start_method<br>
try:<br>
set_start_method(‘spawn’)<br>
except:<br>
pass</p>
</blockquote>
<p>and send the model to device</p>
<blockquote>
<p>device = torch.device(“cuda” if torch.cuda.is_available() else “cpu”)<br>
model = ActorCritic(params).to(device)</p>
</blockquote>
<p>and running the workers by:</p>
<pre><code>            for rank in range(params.num_processes):
                try:
                    p = Process(target=train, args=(rank, params, model, optimizer, indices, sc, device, filename))
                    jobs.append(p)
                    p.start()              
                except Exception as e:
                    print(e)
                    traceback.print_exc()
                    var = traceback.format_exc()
                    f.write("exception:\n"+str(var))
            for p in jobs:
                p.join()
</code></pre>
<p>error stacktrace:</p>
<blockquote>
<p>p.start()<br>
File “C:\Users\Granth\anaconda3\lib\multiprocessing\process.py”, line 112, in start<br>
self._popen = self._Popen(self)<br>
File “C:\Users\Granth\anaconda3\lib\multiprocessing\context.py”, line 223, in _Popen<br>
return _default_context.get_context().Process._Popen(process_obj)<br>
File “C:\Users\Granth\anaconda3\lib\multiprocessing\context.py”, line 322, in _Popen<br>
return Popen(process_obj)<br>
File “C:\Users\Granth\anaconda3\lib\multiprocessing\popen_spawn_win32.py”, line 89, in <strong>init</strong><br>
reduction.dump(process_obj, to_child)<br>
File “C:\Users\Granth\anaconda3\lib\multiprocessing\reduction.py”, line 60, in dump<br>
ForkingPickler(file, protocol).dump(obj)<br>
File “C:\Users\Granth\anaconda3\lib\site-packages\torch\multiprocessing\reductions.py”, line 240, in reduce_tensor<br>
event_sync_required) = storage.<em>share_cuda</em>()<br>
RuntimeError: cuda runtime error (801) : operation not supported at …\torch/csrc/generic/StorageSharing.cpp:247</p>
</blockquote>
<blockquote>
<p>Traceback (most recent call last):<br>
File “”, line 1, in <br>
File “C:\Users\Granth\anaconda3\lib\multiprocessing\spawn.py”, line 105, in spawn_main<br>
exitcode = _main(fd)<br>
File “C:\Users\Granth\anaconda3\lib\multiprocessing\spawn.py”, line 115, in _main<br>
self = reduction.pickle.load(from_parent)<br>
EOFError: Ran out of input</p>
</blockquote>
<p>I am not sure what am I missing here.Also on online reading some posts say that you can not share cuda tensor.</p>
<p>If this is the case then how do we do multiprocessing with cuda</p>

mostafa_said: <p>hi ,<br>
did u solved it</p>
AoT compiler error: a leaf Variable that requires grad is being used in an in-place operation

jimypbr: <p>I’m experimenting with the AoT compiler <a href="https://pytorch.org/functorch/stable/aot_autograd.html" class="inline-onebox" rel="noopener nofollow ugc">functorch.compile (experimental) — functorch 0.2.1 documentation</a> and I’m trying to see if it can capture the forwards, backwards, and optimizer graphs. (pytorch v1.12 on cpu)</p>
<p>My code is:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
from functorch.compile import aot_function


class MyModule(nn.Module):
    def __init__(self):
        super().__init__()
        self.ln1 = nn.Linear(32, 32)
        self.relu = nn.ReLU()
        self.loss = nn.MSELoss()
        
    def forward(self, x):
        x = self.relu(self.ln1(x))
        return x
    
model = MyModule().train()
optimizer = torch.optim.SGD(model.parameters(), lr=1e-4)
criterion = nn.MSELoss()

# This will print the full foward backwards graph!
def training_iter_fn(x, y):
    pred = model(x)
    loss = criterion(pred, y)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()
    return loss

def compiler_fn(fx_module: torch.fx.GraphModule, example_inputs):
    print(fx_module.code)
    print(example_inputs)
    return fx_module

# Pass on the compiler_fn to the aot_function API
aot_print_fn = aot_function(training_iter_fn, fw_compiler=compiler_fn, bw_compiler=compiler_fn)

x = torch.randn(1, 32)
y = x**2 / 5.
loss = training_iter_fn(x, y)
print(loss)
</code></pre>
<p>It works with just the forwards and backwards. However when I include the optimizer step/zero_grad it successfully prints the graph but then throws and exception of:</p>
<pre><code class="lang-auto">File &lt;eval_with_key&gt;.111:29, in forward(self, arg0_1, arg1_1)
     27 empty = torch.ops.aten.empty.memory_format([280], dtype = torch.uint8, device = device(type='cpu'))
     28 _param_constant0_1 = self._param_constant0
---&gt; 29 add__2 = torch.ops.aten.add_.Tensor(_param_constant0_1, add__1, alpha = -0.0001);  _param_constant0_1 = None
     30 _param_constant1_1 = self._param_constant1
     31 add__3 = torch.ops.aten.add_.Tensor(_param_constant1_1, add_, alpha = -0.0001);  _param_constant1_1 = None

File ~/local/anaconda3/envs/dynamo/lib/python3.8/site-packages/torch/_ops.py:257, in OpOverload.__call__(self, *args, **kwargs)
    256 def __call__(self, *args, **kwargs):
--&gt; 257     return self._op(*args, **kwargs or {})

RuntimeError: a leaf Variable that requires grad is being used in an in-place operation.
</code></pre>
<p>The <code>add__2</code> node looks like the weight update in the optimizer.</p>
<p>Anyone have an idea why this is being thrown? Thanks. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Question about LPIPS metric

Yolkandwhite: <p>I read few articles about Learned Perceptual Image Patch Similarity (LPIPS) metric.</p>
<p>In some articles it says that the lower is better. but some articles say that higher is better.</p>
<p>which one is correct??</p>

mahmood: <p>In some cases, higher LPIPS are suitable for evaluating the diversity of images generated by GANs, but if we are generating images or comparing images to see how different the original images are from the ones to be compared, lower LPIPS are suitable for that as well.</p>
About the NVFUSER

JeeLee: <p>Afer reading <a href="https://pytorch.org/tutorials/intermediate/nvfuser_intro_tutorial.html" rel="noopener nofollow ugc"> ACCELERATE YOUR SCRIPTS WITH NVFUSER</a>, we  began to test the acceleration effect of nvfuser.In some networks such as ResNet50，DenseNet，torchscript can achieve ～2.x speedup(vs eager mode),but when we tested in <a href="https://github.com/open-mmlab/mmediting/blob/master/mmedit/models/backbones/sr_backbones/basicvsr_net.py#L15" rel="noopener nofollow ugc">BascicVSR</a>,  we did’t see any speedup,why?</p>

JeeLee: <p>Our script:</p>
<pre><code class="lang-python">import os
import time
import torch
import torch.nn as nn
import torch.nn.functional as F
from mmedit.models.registry import BACKBONES
from torchvision import models
@torch.no_grad()
def profile_workload_nn(forward_func,inputs, iteration_count=100, label=""):
    
    # Perform warm-up iterations
    for _ in range(20):
        # Run model, forward and backward
        output = forward_func(inputs)

    
    # Synchronize the GPU before starting the timer
    torch.cuda.synchronize()
    start = time.perf_counter()
    for _ in range(iteration_count):
        # Run model, forward and backward
        output = forward_func(inputs)


    # Synchronize the GPU before stopping the timer
    torch.cuda.synchronize()
    stop = time.perf_counter()
    iters_per_second = iteration_count / (stop - start)
    if label:
        print(label)
    print("Average iterations per second: {:.2f}".format(iters_per_second))
def test_nvfuser():
    eager_module = BACKBONES.get('BasicVSRNet')()
    # eager_module=models.resnet50(pretrained=False)
    # eager_module=models.densenet121(pretrained=False)
    eager_module.half().cuda()
    input1 = torch.ones(1, 5,3, 512, 512).half().cuda()
    eager_module.eval()
    with torch.no_grad():
        eager_result = eager_module(input1)

        profile_workload_nn(
        eager_module, input1, iteration_count=100, label="Eager")
        trace_module = torch.jit.trace(eager_module,input1)
        profile_workload_nn(
        trace_module, input1, iteration_count=100, label="TorchScript")
        trace_result=trace_module(input1)
        error=torch.abs(eager_result-trace_result).mean()
        print(error)

if __name__ == "__main__":
    test_nvfuser()
 

</code></pre>

ptrblck: <p>The observed speedup depends on the model architecture and in particular which operations are used. In the last stable release (PyTorch 1.12.0) nvFuser was targeting pointwise, reduction, and normalization operations. To see the latest development install the latest nightly binary and rerun your scripts.</p>

JeeLee: <p>Thanks for your reply,  our pytorch version is <code>1.12.1+cu116</code> ,and GPU is <code>RTX 3090 Ti</code>. We also printed the profiling info refered to <a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/jit/codegen/cuda/README.md" rel="noopener nofollow ugc">NVFUSER</a>, the fused kernel is indeed generated, but it does not accelerate.</p>
Resizing Tensors in Pytorch Android

diningeachox: <p>Hi everyone,</p>
<p>I just trained a model on python and used torch.jit.trace to serialize it for usage in android development. One important detail is that I would need to resize the raw data I’m getting from the android app into the input dimensions used by my model. What is the syntax/API required to resize tensors in android?</p>
<p>I’ve been following the tutorial here: <a href="https://pytorch.org/mobile/android/" class="inline-onebox" rel="noopener nofollow ugc">Android | PyTorch</a></p>
<p>The line</p>
<pre><code class="lang-auto">Tensor inputTensor = TensorImageUtils.bitmapToFloat32Tensor(bitmap,
    TensorImageUtils.TORCHVISION_NORM_MEAN_RGB, TensorImageUtils.TORCHVISION_NORM_STD_RGB);
</code></pre>
<p>is used to convert an image into a float32 tensor, but I couldn’t find anything about resizing or transformations on that page.</p>

lllllllt: <p>Have u solved the problem?<br>
I’m just meet the problem cuz my model needs input in size((1,1,h,w))<br>
while PyTorch only provide function that support 3 channels(RGB)</p>
How to replicate a list for DataParallel

pytorcher: <p>My module includes a list that should be kept separate for the two replicas in my DataParallel.  I essentially want to do this exact thing but with a list instead of a single value:  <a href="https://github.com/pytorch/pytorch/issues/2864" class="inline-onebox" rel="noopener nofollow ugc">DataParallel doesn't replicate module's member variables · Issue #2864 · pytorch/pytorch · GitHub</a>.</p>
Using AMP with CUDA Graphs and DDP - Scaler Error

clarky: <p>I am investigating speeding up training using the CUDA graphs pytorch API. My training loop functions correctly with AMP/autocast disabled but with it enabled, I get an error on the scaler.step(optimiser) instruction.</p>
<p>Below is my training loop:</p>
<pre><code class="lang-auto">warmup_loader = get_warmup_loader(args, rank)
device = get_device(args)
use_amp = args.get('amp').get('enabled')

if is_main_node(rank):
    print(f'\nWarming up')

# CUDA Graphs Warming Up Iteration

s = torch.cuda.Stream()
s.wait_stream(torch.cuda.current_stream())
with torch.cuda.stream(s):
    model = get_model(args)
    model.train()

    for batch, (images, labels) in enumerate(tqdm(warmup_loader, disable=not is_main_node(rank))):
        images = images.to(device, memory_format=get_memory_format(args))
        labels = labels.to(device, memory_format=get_memory_format(args))

        with autocast(enabled=use_amp):
            y_pred = model(images)['out']
            loss = criterion(y_pred, labels)

        if use_amp:
            scaler.scale(loss).backward()
            scaler.step(optimizer)
            scaler.update()
        else:
            loss.backward()
            optimizer.step()

        optimizer.zero_grad(set_to_none=True)

torch.cuda.current_stream().wait_stream(s)

# CUDA Graphs Capture

capture_input = torch.empty(images.shape, device=device)
capture_target = torch.empty(labels.shape, device=device)

g = torch.cuda.CUDAGraph()
optimizer.zero_grad(set_to_none=True)
with torch.cuda.graph(g):
    with autocast(enabled=use_amp):
        capture_y_pred = model(capture_input)['out']
        capture_loss = criterion(capture_y_pred, capture_target)

    if use_amp:
        scaler.scale(capture_loss).backward()
    else:
        capture_loss.backward()
        optimizer.step()

if is_main_node(rank):
    print("\nTraining")

# Training on real data

start = time.time()
for epoch in range(args.get('hyper-params').get('epochs')):
    for batch, (images, labels) in enumerate(tqdm(train_loader, disable=not is_main_node(rank))):
        capture_input.copy_(images)
        capture_target.copy_(labels)
        g.replay()
        if use_amp:
            scaler.step(optimizer)
            scaler.update()
end = time.time()

if is_main_node(rank):
    print(f"\nTraining took: {end - start:.2f}s")
</code></pre>
<p>I get the following error:</p>
<pre><code class="lang-auto">assert len(optimizer_state["found_inf_per_device"]) &gt; 0, "No inf checks were recorded for this optimizer."
AssertionError: No inf checks were recorded for this optimizer.
</code></pre>
<p>I have been following the documentation found here: <a href="https://pytorch.org/docs/master/notes/cuda.html#usage-with-torch-cuda-amp" class="inline-onebox" rel="noopener nofollow ugc">CUDA semantics — PyTorch master documentation</a>.<br>
From my understanding the scaler/optimiser should know whether to skip or not based on inf/NaN?</p>
<p>Any help would be great.</p>

clarky: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Could you help with this issue?</p>
How can I put the model on my CUDA device in LibTorch?

Alio: <p>I am not able to put my the model I’ve written in LibTorch on my CUDA device. I’m sure the CUDA device is working because I can put tensors on it.</p>
<p>I tried putting the copies of the model I stored on the CUDA device like so,</p>
<pre><code>auto actor = Act_Net(low, high);
auto target_actor = Act_Net(low, high);
auto critic = Critic_Net();
auto target_critic = Critic_Net();

actor-&gt;to(device);
target_actor-&gt;to(device);
critic-&gt;to(device);
target_critic-&gt;to(device);  
</code></pre>
<p>This returns the error…</p>
<pre><code>/home/iii/tor/m_gym/multiv_normal.cpp:183:1: error: ‘actor’ does not name a type
  183 | actor-&gt;to(device);
      | ^~~~~
/home/iii/tor/m_gym/multiv_normal.cpp:184:1: error: ‘target_actor’ does not name a type
  184 | target_actor-&gt;to(device);
      | ^~~~~~~~~~~~
/home/iii/tor/m_gym/multiv_normal.cpp:185:1: error: ‘critic’ does not name a type
  185 | critic-&gt;to(device);
      | ^~~~~~
/home/iii/tor/m_gym/multiv_normal.cpp:186:1: error: ‘target_critic’ does not name a type
  186 | target_critic-&gt;to(device);
      | ^~~~~~~~~~~~~
</code></pre>
<p>This is what it looks like in VS Code.</p>
<p><a href="https://i.stack.imgur.com/JbOdo.png" rel="noopener nofollow ugc"><img src="https://i.stack.imgur.com/JbOdo.png" alt="enter image description here" width="602" height="283"></a></p>
<p>The <a href="https://pytorch.org/tutorials/advanced/cpp_frontend.html" rel="noopener nofollow ugc">LibTorch documentation</a> says…</p>
<p>“and also our model parameters should be moved to the correct device:”</p>
<pre><code>generator-&gt;to(device);
discriminator-&gt;to(device);
</code></pre>
<p>Maybe I don’t know where in the script to put <code>actor-&gt;to(device);</code>?</p>
How do I finetune a model while preserving layer names

venuv: <p>When I fine tune a pretrained resnet152 model, I seem to lose all the named layers I’d like access to. I’ve include the simple fine tuned model, and the named layers of both pretrained and fine tuned. TIA.</p>
<blockquote>
<p>class ConvNet3(nn.Module):</p>
<pre><code> def __init__(self):
    super().__init__()
    model = models.resnet152(pretrained=True)
    model.fc = nn.Linear(2048, 10)
    self.model = model
    
def forward(self, x):        
    return self.model(x) # [batch_size, 10]
</code></pre>
</blockquote>
<p>The named layer printouts as follows:</p>
<blockquote>
<p>import torchvision.models as models<br>
model = ConvNet3().eval()<br>
print([n for n, _ in model.named_children()])</p>
<p>model = models.resnet152(pretrained=True).eval()<br>
print([n for n, _ in model.named_children()])</p>
</blockquote>
<p>Output from above print statements</p>
<blockquote>
<p>[‘model’]<br>
[‘conv1’, ‘bn1’, ‘relu’, ‘maxpool’, ‘layer1’, ‘layer2’, ‘layer3’, ‘layer4’, ‘avgpool’, ‘fc’]</p>
</blockquote>

venuv: <p>TIL that the resnet model gets wrapped in ConvNet3. So ‘model.model.named_children’ in the first print statement would have worked.</p>
HELP! Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

Andrew_Yin: <p>I use nn.Parameter() to create relative_position_index, and it can’t be moved into gpu, the code is attached, and i can’t figure it out.</p>
<blockquote>
<p>Blockquote</p>
</blockquote>
<pre><code class="lang-auto">def __init__(self, ...):
      self.relative_bias_position_table = nn.Parameter(
      torch.zeros((2*window_size[0]-1)*(2*window_size[1]-1), self.num_heads))
      nn.init.trunc_normal_(self.relative_bias_position_table, std=0.02)
      coords = torch.stack(torch.meshgrid(torch.arange(window_size[0]), torch.arange(window_size[1]))) 
      coords_flatten = torch.flatten(coords, 1)  # [2, h*w]
      relative_coords_bias = coords_flatten[:, :, None] - coords_flatten[:, None, :]
      relative_coords_bias[0, :, :] += window_size[0] - 1
      relative_coords_bias[1, :, :] += window_size[1] - 1
      relative_coords_bias[0, :, :] *= relative_coords_bias[1, :, :].max() + 1
      bias_index = relative_coords_bias.sum(0)
      self.register_buffer("bias_index", bias_index)
      self.bias_embedding = self.relative_bias_position_table[self.bias_index.view(-1)].view(self.window_size[0] * self.window_size[1], self.window_size[0] * self.window_size[1], self.num_heads)
      self.bias_embedding = self.bias_embedding.permute(2, 0, 1).contiguous().unsqueeze(0)

def forward(self,x):
     attn = attn + self.bias_embedding

</code></pre>
<pre><code class="lang-auto">File "D:\ProgramFile\anaconda\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "D:\yzw\TEST\model.py", line 130, in forward
    attn = attn + self.bias_embedding
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
</code></pre>

ptrblck: <p>Based on your code snippet it seems you are creating non-leaf tensors by applying differentiable operations on the parameter. Create the tensor first and wrap it into an <code>nn.Parameter</code> afterwards without calling any operations on it afterwards. Once this is fixed <code>model.to()</code> should move this parameter to the device.</p>
GPU ultization low but CPU memory usuage increase after each batch

usr0909: <p>Hi all, I am very new to this, and sorry if I asked a stupid question. I used a model imported from <a href="https://github.com/tding1/Neural-Collapse/tree/main" rel="noopener nofollow ugc">GitHub</a> to train on my own data. The total input is ~300GB and loaded with ‘’‘torchvision.datasets.ImageFolder’‘’. I notice that the CPU memory keeps building up but the GPU utilization is very low, please see the metric below. Is this because my input data is too large to be loaded into CPU? Is there any suggestion to improve the training speed (GPU utilization)? Thanks a lot!<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/f/4fa375dd9bcfce33f918df671f63059afe11daaf.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/4fa375dd9bcfce33f918df671f63059afe11daaf" title="Screen Shot 2022-10-11 at 13.41.10"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/f/4fa375dd9bcfce33f918df671f63059afe11daaf_2_690x425.jpeg" alt="Screen Shot 2022-10-11 at 13.41.10" data-base62-sha1="bmvVOtbyLDcHwHYyJ6W2BxnEeWr" width="690" height="425" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/f/4fa375dd9bcfce33f918df671f63059afe11daaf_2_690x425.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/4/f/4fa375dd9bcfce33f918df671f63059afe11daaf_2_1035x637.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/f/4fa375dd9bcfce33f918df671f63059afe11daaf.jpeg 2x" data-dominant-color="F5F6F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2022-10-11 at 13.41.10</span><span class="informations">1320×814 54.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p><code>ImageFolder</code> will lazily load the samples and won’t preload the entire dataset. Based on the posted screenshot it seems the host memory peaks at ~90GB and stays at this level during the training which might be expected depending on the actual training script. The low GPU utilization might be caused by a bottleneck e.g. in the data loading pipeline. Try to profile your code to narrow down the bottleneck and e.g. try to increase the number of workers in case the data loading could benefit from it.</p>
Strange behavior with PyTorch when setting tensor values by index

meower: <p>I have a non-zero tensor<br>
<code>input</code> with shape <code>torch.Size([128, 10])</code> and a zero tensor with identical size called <code>z</code>. I’m trying to access <code>z</code> by index and set it accordingly as:</p>
<pre><code class="lang-auto">N, C = input.shape
z = torch.zeros(input.shape)

for n in range(N):
    for c in range(C):
        if (c == target[n]):
            z[n, c] = input[n, c]
        else:
            z[n, c] = -1 * input[n, c]

</code></pre>
<p>where <code>target</code> is another non-zero tensor with shape <code>torch.Size([128])</code>. When I run this, only the very fist element of <code>z</code> gets updated and the rest are all zeroes. What am I doing wrong?</p>

ptrblck: <p>This code works for me and all values in <code>z</code> are set:</p>
<pre><code class="lang-python">input = torch.randn(128, 10)

N, C = input.shape
z = torch.zeros(input.shape)
target = torch.randint(0, C, (N,))

for n in range(N):
    for c in range(C):
        if (c == target[n]):
            z[n, c] = input[n, c]
        else:
            z[n, c] = -1 * input[n, c]

(z==0.).float().sum()
# tensor(0.)
</code></pre>

meower: <p>Hmm, strange, I wonder what I am doing wrong. Maybe it is related to the allocation of input/target to MPS since I am running it on Macbook Pro with M1.</p>

ptrblck: <p>That could be the case as I’m running on a plain Linux node. In case you are seeing the issue in the latest nightly, could you create an issue on GitHub, please?</p>

meower: <p>This has been fixed in <a href="https://github.com/pytorch/pytorch/commit/8da704cdb7f68bfa09516e7be17f004b98c48eb3" class="inline-onebox" rel="noopener nofollow ugc">[MPS] Remove incorrect asserts from `Copy.mm` (#86184) · pytorch/pytorch@8da704c · GitHub</a>.</p>
Torch-TensorRT + FX Frontend with torch.fx.wrap

collinmccarthy: <p>Hi,</p>
<p>I have a workflow that uses torch.fx and it works great. It did, however, require wrapping quite a few functions in <code>torch.fx.wrap</code> to declare them as leafs to not be traced through (for dynamic control flow, specifically for Swin-Transformer with different input sizes that pads various things to work with any input size). What is the implication of this when I try to create a TensorRT version of the model via the <a href="https://pytorch.org/TensorRT/tutorials/getting_started_with_fx_path.html" rel="noopener nofollow ugc">Torch-TensorRT FX Frontend</a>?</p>
<p>Specifically,</p>
<ol>
<li>
<p>Will Torch-TensorRT still be able to perform optimizations for code wrapped with <code>torch.fx.wrap</code> or will it split the graph and force torch to execute those portions?</p>
</li>
<li>
<p>Will the output be any different than if I use the JIT scripting frontend instead of FX, and then pass this scripted model to Torch-TensorRT, so I don’t have to worry about “wrapped”/skipped functions? I don’t want to bother adding this workflow if Torch-TensorRT is just going to do this for me internally, by scripting the FX version (if that would even expose the wrapped function internals).</p>
</li>
</ol>
<p>Thank you!<br>
-Collin</p>
How to use CRF in Pytorch

Shreya_Arora: <p>I want to convert the following keras code to pytorch:</p>
<p>crf_layer = CRF(units=TAG_COUNT)<br>
output_layer = crf_layer(model)</p>
<p>I was trying the following:<br>
crf_layer = self.crf()<br>
output_layer = crf_layer(x)</p>
<p>But I am getting the following error:<br>
crf_layer = self.crf()<br>
*** TypeError: forward() missing 2 required positional arguments: ‘emissions’ and ‘tags’</p>
Best way of multi-dimensional tensor indexing

Dmitry: <p>Hello,</p>
<p>What is the best way to replace values of specific indices in a multi-dimensional tensor, given a smaller tensor of new values with a fewer number of dimensions, while not ruining the gradient flow?</p>
<p>So far I did it like that:</p>
<pre><code class="lang-auto">input = torch.rand(2, 3, 4, 4)

top_k_ind = torch.topk(input [:, :, 0, :], 5, largest=True)[1] # indeces only

for dim_0 in range(top_k_ind.shape[0]):
    for dim_1 in range(top_k_ind.shape[1]):
        input [dim_0, dim_1, 0, top_k_ind[dim_0 , dim_1 , :]] = 0 # 3d dim of input is not involved
</code></pre>
<p>However, I doubt this is the right way of doing it in PyTorch.</p>
<p>I think there must be a solution like that:</p>
<pre><code class="lang-auto">input[ top_k_ind[0], top_k_ind[1], 0, top_k_ind[2] ] = 0 # 0, 1, 2 - dimensions
</code></pre>
<p>Thank everyone in advance.</p>
Switching between FSDP and DDP

rowhanm: <p>Thanks for all the help with and building FSDP. I’m able to successfully train my models with FSDP but there are some operations during my evaluation that don’t quite work with FSDP and I need to use DDP. I wanted to check if there is a way I can “unshard”/reset my model after training to run evaluation (assuming I am able to fit my model in memory without FSDP in no_grad() mode)</p>
<p>A general flow would look like:</p>
<pre><code class="lang-auto">model = FSDP(model,...)
train(model)
.
.
reset_model() # this would basically remove the FSDP wrapper
model = DDP(model,...)
evaluate(model)
</code></pre>
<p>Thanks!</p>

agu: <p>We should figure out and document the canonical way to do this, but here is something that can hopefully unblock you for now:</p>
<pre><code class="lang-auto">from torch.distributed.fsdp import FullyShardedDataParallel as FSDP
from torch.distributed.fsdp import FullStateDictConfig, StateDictType

def reset_model(fsdp_model: FSDP, model_ctor: Callable):
    with FSDP.state_dict_type(
        fsdp_model,
        StateDictType.FULL_STATE_DICT,
        FullStateDictConfig(rank0_only=True, offload_to_cpu=True),
    ):
        # Since `rank0_only=True`, only rank 0 has a full state dict in memory
        state_dict = fsdp_model.state_dict()

    # Construct a nonwrapped model like what was passed into FSDP, where the
    # parameter/buffer initialization does not matter
    nonwrapped_model = model_ctor()  # on CPU
    if rank == 0:
       nonwrapped_model.load_state_dict(state_dict)
    # Constructing DDP will broadcast parameters/buffers from rank 0 to all ranks
    ddp_model = DDP(nonwrapped_model.to(rank), device_ids=[rank])
</code></pre>
<p><code>model_ctor()</code> can initialize on GPU, in which case <code>state_dict</code> should be saved on GPU (with <code>offload_to_cpu=False)</code>.</p>
<p>I wrote this off the top of my head and have not tested it. However, it conveys the general idea: You save a full state dict and load it into a nonwrapped version of the module before wrapping with DDP. DDP will broadcast module states in its constructor, so that saves some additional logic.</p>
Padding=1 and "same" give different result

ximing: <p>Based on what I know, in the Conv2D, padding has two value: 0 and 1. 0 equals to “valid” which is no padding while 1 equals to “same” which means add 0 as padding and make the output size the same as input size.However, when I tried using ''valid" and “same” in a 2D convolutional layer passing a input (3<em>64</em>64), I found their output sizes are the same. In theory, I think the output size with no padding should be smaller than 64<em>64? I also when I set the parameter to 1, the output size becomes 66</em>66. I think this looks very strange to me. Does anyone have any idea?</p>
<pre><code class="lang-auto">sample_torch = sample.reshape(1,3,64,64)
conv1 = nn.Conv2d(3, 64, 1, stride=1, padding=0).float()
y = conv1(torch.tensor(sample_torch).float())
print(y.shape)
conv2 = nn.Conv2d(3, 64, 1, stride=1, padding=1).float()
y = conv2(torch.tensor(sample_torch).float())
print(y.shape)
conv1 = nn.Conv2d(3, 64, 1, stride=1, padding="valid").float()
y = conv1(torch.tensor(sample_torch).float())
print(y.shape)
conv2 = nn.Conv2d(3, 64, 1, stride=1, padding="same").float()
y = conv2(torch.tensor(sample_torch).float())
print(y.shape)
</code></pre>
<p>This is the output</p>
<pre><code class="lang-auto">torch.Size([1, 64, 64, 64])
torch.Size([1, 64, 66, 66])
torch.Size([1, 64, 64, 64])
torch.Size([1, 64, 64, 64])
</code></pre>

ptrblck: <aside class="quote no-group" data-username="ximing" data-post="1" data-topic="163584">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/x/3bc359/48.png" class="avatar"> ximing:</div>
<blockquote>
<p>Based on what I know, in the Conv2D, padding has two value: 0 and 1. 0 equals to “valid” which is no padding while 1 equals to “same” which means add 0 as padding and make the output size the same as input size.</p>
</blockquote>
</aside>
<p>That’s not generally true since the padding argument accepts different values and the “same” or “valid” option also depends on the kernel size, the stride, and other arguments.<br>
In your use case you are using a kernel size of 1 which yields the same output size without any padding.</p>
How to run fairest-train with Mac M1 Max GPU

Haifa_Tl: <p>Hi,<br>
I need a little help. I have been trying to run fairseq-train on my Mac M1 Max using the GPU, but it is running on CPU.</p>
<p>torch.backends.mps.is_available() gives “True”</p>
<p>How can I force fairest-train to use “mps”?</p>
<p>!fairseq-train /Users/ha/data/preprocessed --max-epoch 7<br>
–encoder-normalize-before --decoder-normalize-before<br>
–arch transformer --layernorm-embedding<br>
–task translation_multi_simple_epoch<br>
–sampling-method “temperature”<br>
–sampling-temperature 1.5<br>
–encoder-langtok “src”<br>
–decoder-langtok<br>
–lang-pairs “$LANGPAIRS”<br>
–criterion label_smoothed_cross_entropy --label-smoothing 0.2<br>
–optimizer adam --adam-eps 1e-06 --adam-betas ‘(0.9, 0.98)’<br>
–lr-scheduler inverse_sqrt --lr 3e-04 --warmup-updates 2500 --max-update 40000<br>
–dropout 0.3 --attention-dropout 0.1 --weight-decay 0.0<br>
–max-tokens 1024 --max-tokens-valid 1024 --update-freq 2<br>
–save-interval 1 --save-interval-updates 5000 --keep-interval-updates 10 --no-epoch-checkpoints<br>
–seed 222 --log-format simple --log-interval 50 --ddp-backend=legacy_ddp</p>
Store model parameters on gpus

Tianyu9748: <p>Is there any Lib helps to store the parameters of the model on gpu’s global memory (or on host memory), instead of the disk.<br>
I have read the <code>torch.save</code>, it seems like will save the model parameters on disk.</p>
<p>Best<br>
Max</p>

ptrblck: <p>The <code>state_dict</code> which you would store to your disk via <code>torch.save</code> is already in the host or GPU RAM (it has to live somewhere before you can serialize it) so could you explain your use case a bit more, please?</p>

Tianyu9748: <p>Hi ptrblck,<br>
Sorry for the unclear explanation, hope this time I could make it clear.</p>
<p>My aim:<br>
After finishing the training of a model, for example Resnet512, could I save the updated parameters to somewhere on the global memory?</p>
<ol>
<li>If we could save parameters to somewhere on the global memory, could we load these parameters to set up a new Resnet512?</li>
<li>If we could save parameters to somewhere on the global memory, is there anyway to keep it ‘alive’, even after finishing the training and stop the training process?</li>
</ol>
<p>Best<br>
Max</p>

ptrblck: <ol>
<li>
<p>The parameters are already stored in the global memory of your GPU assuming you are training the model on the GPU. If you want to create a copy, you could use <code>copy.deepcopy</code> to do so.</p>
</li>
<li>
<p>No, you cannot use the GPU’s global memory for serialization and the memory will be released after your Python process exits.</p>
</li>
</ol>

Tianyu9748: <p>Thank you for the explanation. That helps a lot. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
How to tell if AMP is enabled in Aten

alugorey: <p>I’m trying to add some functionality to the Convolutional code but I only want it to run if AMP is enabled. I attempted to use ATen/autocast_mode.h::is_enabled() but this flag doesn’t change whether I use a model with autocasting or not. It seems to always return false. Is there a way to query if autocasting is enabled from within the C++ code in the ATen sub component?</p>

ptrblck: <p><code>aten::is_autocast_enabled()</code> should work in libtorch.</p>

alugorey: <p>Do you happen to know what the include statement is for that? i got  this error</p>
<blockquote>
<p>pytorch/aten/src/ATen/native/miopen/Conv_miopen.cpp:767:39: error: ‘aten’ has not been declared<br>
767 |   std::cout &lt;&lt; "autocastEnabled: " &lt;&lt; aten::is_autocast_enabled() &lt;&lt; std::endl;</p>
</blockquote>
<p>To clarify, I am trying to modify code within pytorch.</p>

alugorey: <p>Do you happen to know what the include statement is for that? i got this error:</p>
<blockquote>
<p>pytorch/aten/src/ATen/native/miopen/Conv_miopen.cpp:768:45: error: ‘is_autocast_enabled’ is not a member of ‘c10::aten’<br>
768 |   std::cout &lt;&lt; "autocastEnabled: " &lt;&lt; aten::is_autocast_enabled() &lt;&lt; std::endl;</p>
</blockquote>
<p>To clarify,I am trying to modify code within pytorch.</p>

ptrblck: <p>I might be wrong and the posted method could be JIT-specific. Could you check <code>at::autocast::is_enabled()</code> instead (I’m not in front of my workstation as I would confirm it otherwise myself).</p>

alugorey: <p>I tried that as well. While it did return an actual value, that value was false whether I used a model the with autocasting or not.</p>
Libtorch, does it support visual studio 2008?

Hisa: <p>Hello everyone,<br>
I got a question when I use the libtorch in C++ code.<br>
I succeeded in building my C++ code using visual studio 2019 but failed using visual studio 2008. I think it is because of different compiler versions.</p>
<p><strong>Does the libtorch support visual studio 2008?</strong><br>
<strong>If not, are there other methods to import the model into C++ using visual studio 2008?</strong></p>
<p>Any help would be much appreciated. Thanks!</p>

wter26: <p>I think its not a good idea, cause some code need c++17 or higher stardard to support.Its lots of work to trans the code to adapt to the vs 2008.Trust me.</p>
PIL.UnidentifiedImageError in Dataloader

Izu97: <pre><code class="lang-auto">def load_data():
    ImageFile.LOAD_TRUNCATED_IMAGES = True

    device = torch.device('cuda:0') if torch.cuda.is_available() else torch.device('cpu')
    print("Running on {}".format(device))
    data_transform = transforms.Compose([
            transforms.RandomResizedCrop(crop),
            transforms.RandomHorizontalFlip(),
            transforms.ToTensor(),
            transforms.Normalize(mean=mean,
                                std=std)
        ])
    dataset = datasets.ImageFolder(root=current_dir,
                                   transform=data_transform
                                   )

    print("Size of dataset is ",len(dataset))

    Ntrain = int(len(dataset) * cutoff)
    dataset = shuffle(dataset)
    train_data, test_data = dataset[:Ntrain], dataset[Ntrain:]

    train_loader = torch.utils.data.DataLoader(train_data,
                                                batch_size=batch_size,
                                                shuffle=True,
                                                num_workers=workers
                                                ).to(device)

    test_loader  = torch.utils.data.DataLoader(test_data,
                                                batch_size=batch_size,
                                                shuffle=True,
                                                num_workers=workers
                                                ).to(device)
    return train_loader, test_loader
</code></pre>
<p>after I was loading data into DataLoader it was generated the following error. In my dataset I had currupted image named 666.jpg. How to fix this issue.Thank you.</p>
<p>error msg:<br>
“cannot identify image file %r” % (filename if filename else fp)<br>
PIL.UnidentifiedImageError: cannot identify image file &lt;_io.BufferedReader name=‘E:\My Projects\Cat-vs-Dog-Image-Classifier-Differnet-Models-with-Pytroch\Data\Cat\666.jpg’&gt;</p>

ptrblck: <p>Could you check, if the file is corrupt, i.e. can you open it with another viewer?<br>
If that’s the case, you would either have to download it again or remove it.</p>

Izu97: <p>Thank you. I was manually checked the dataset and I just found only 2 courrupted images. after removing them works fine</p>

Wenuka_Gunarathna: <p>is there a way to ignore corrupted images (in dataset or in the dataloader), instead of manually removing them?</p>

ambujpawar: <p>Hi, yeah this can be done using <code>is_valid_file </code>  argument while using <code>ImageFolder</code> as well others datasets loaders inheriting <code>DatasetFolder</code>.</p>
<p>The documentation states that:</p>
<blockquote>
<p>is_valid_file (optional): A function that takes path of a file<br>
and checks if the file is a valid file<br>
(used to check of corrupt files) both extensions and<br>
is_valid_file should not be passed. Defaults to None.</p>
</blockquote>
<p>For more information, look into <a href="https://pytorch.org/vision/stable/generated/torchvision.datasets.ImageFolder.html" rel="noopener nofollow ugc">this</a></p>
How to slice multiple interval over tensor

jhtorch: <p>Hi, is there any method to do the same thing with the code below without loop:</p>
<p>tensors have the dimension [B, C, T, H, W]<br>
I want to slice the tensor in T axis with multiple start, end pair</p>
<pre><code class="lang-auto">   for b in range(B):
        for h in range(H):
            for w in range(W):
                interval = torch.arange(starts[b][h][w],ends[b][h][w])
                cropped[b,:,:,h,w] = origin[b, :, interval, h,w]
</code></pre>
How to use torch.distributions.multivariate_normal.MultivariateNormal in multi-gpu mode

Skyler_Huang: <p>In single gpu mode,MultivariateNormal can run correctly, but when i switch to multi-gpu mode, always get the error:</p>
<p>G = torch.exp(m.log_prob(Delta))<br>
File “xxxxx”, line 210, in log_prob<br>
M = _batch_mahalanobis(self._unbroadcasted_scale_tril, diff)<br>
File “xxxxx”, line 57, in _batch_mahalanobis<br>
M_swap = torch.triangular_solve(flat_x_swap, flat_L, upper=False)[0].pow(2).sum(-2) # shape = b x c<br>
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasStrsmBatched( handle, side, uplo, trans, diag, m, n, alpha, A, lda, B, ldb, batchCount)</p>
<p>If use gpu MultivariateNormal must specify device, so I specify the device in forward function:<br>
the code is:</p>
<p>Delta # is a random tensor<br>
mean = torch.zeros(2).to(x.device)<br>
cov = torch.eye(2).to(x.device)<br>
m = MultivariateNormal(mean, cov * self.sigma**2)<br>
G = torch.exp(m.log_prob(Delta))</p>
<p>I would be very grateful if you could give some suggestions</p>

Julieee: <p>Hi Skyler,</p>
<p>I encountered the same problem. Did you figure out the solution?</p>
<p>Thanks,<br>
Julie</p>

Chaojie_FU: <p>I came into this problem recently with CUDA version <code>11.5</code> and driver version <code>495.29.05</code> and it seems that <strong>MultivariateNormal on GPU cannot deal with covariance matrix’s shapes correctly</strong>. For example, the code below don’t work:</p>
<pre><code class="lang-python">num = 524288
loc = torch.rand(num, 12, device="cuda:0")
cov = torch.diag(torch.ones(12, device="cuda:0"))
actions = torch.rand(num, 12, device="cuda:0")
dist = MultivariateNormal(loc, cov)

dist.log_prob(actions)
</code></pre>
<p>the code above will report error as below:</p>
<pre><code class="lang-bash">Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "xxxxxx/lib/python3.8/site-packages/torch/distributions/multivariate_normal.py", line 210, in log_prob
    M = _batch_mahalanobis(self._unbroadcasted_scale_tril, diff)
  File "xxxxxx/lib/python3.8/site-packages/torch/distributions/multivariate_normal.py", line 57, in _batch_mahalanobis
    M_swap = torch.triangular_solve(flat_x_swap, flat_L, upper=False)[0].pow(2).sum(-2)  # shape = b x c
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasStrsmBatched( handle, side, uplo, trans, diag, m, n, alpha, A, lda, B, ldb, batchCount)`
</code></pre>
<p>The problem solved when I specify the tensor <code>mat</code> 's shape manually so that <code>loc</code> and <code>mat</code> have the same batch_size:</p>
<pre><code class="lang-python">num = 524288
loc = torch.rand(num, 12, device="cuda:0")
cov = torch.diag(torch.ones(12, device="cuda:0")).unsqueeze(0).repeat(num, 1, 1)
actions = torch.rand(num, 12, device="cuda:0")
dist = MultivariateNormal(loc, cov)

dist.log_prob(actions)
</code></pre>
<p>It might be the same problem as the one that you encountered.</p>