akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
What is "_dedup_tensors.py:44 INFO p:SpawnProcess-1 t:MainThread: Duplicate keys to remove"

username: gateofsoul
cooked: <p>Hi. I’m practicing PyTorch FSDP coding these days, and trying to make save and load snapshots for it.</p>
<p>I ran the example code(torch/distributed/checkpoint/examples/fsdp_checkpoint_example.py) and it works well. However, every time I ran it, an warning message is always showed as</p>
<p><code>_dedup_tensors.py:44 INFO p:SpawnProcess-1 t:MainThread: Duplicate keys to remove:</code></p>
<p>at</p>
<pre><code class="lang-auto">dist_cp.save_state_dict(
            state_dict=state_dict,
            storage_writer=dist_cp.FileSystemWriter(CHECKPOINT_DIR),
        )
</code></pre>
<p>What does it means and how can I fix it?</p>
Can't forward pass under colab but can under linux

username: cltexe
cooked: <p>Title could be vague but here is the problem:<br>
I have a <code>codes = net.encoder(x)</code> line which performs a single pass on encoder with its forward function defined as</p>
<blockquote>
<p>def forward(self, x):</p>
<pre><code>    x = self.input_layer(x)
</code></pre>
</blockquote>
<p>and <code>self.input_layer</code> is defined as</p>
<blockquote>
<p>self.input_layer = Sequential(Conv2d(3, 64, (3, 3), 1, 1, bias=False),<br>
BatchNorm2d(64),<br>
PReLU(64))</p>
</blockquote>
<p>apparently on my linux machine with torch 2.0.1 this runs fine but in colab, it doesn’t throws error but can’t pass beyond <code>x = self.input_layer(x)</code> which i checked with a print statement afterwards. How to debug this?</p>

username: ptrblck
cooked: <p>You could try to reproduce the hang in <code>ipython</code> in a terminal and in case you are able to do so, you could attach <code>gdb</code> to check where the code is hanging for further debugging.</p>

username: cltexe
cooked: <p>I tried on the local in an Ipython and it ran smoothly just like the command line. Colab is problematic and indeed it doesn’t throws error or hangs, acts like it completed command successfully. And the error seems to originate from torch library.</p>
What are the bubbles before `cudaLaunchKernel`?

username: nahso
cooked: <p>There are always bubbles before executing cudaKernelLaunch, eg:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/1/51292ebad0113478c1cfbfca62f7420f1ba1633b.png" data-download-href="https://discuss.pytorch.org/uploads/default/51292ebad0113478c1cfbfca62f7420f1ba1633b" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/1/51292ebad0113478c1cfbfca62f7420f1ba1633b_2_690x95.png" alt="image" data-base62-sha1="bzYUjsgHbyMMnF6o00B8P1qTtCb" width="690" height="95" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/1/51292ebad0113478c1cfbfca62f7420f1ba1633b_2_690x95.png, https://discuss.pytorch.org/uploads/default/original/3X/5/1/51292ebad0113478c1cfbfca62f7420f1ba1633b.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/1/51292ebad0113478c1cfbfca62f7420f1ba1633b.png 2x" data-dominant-color="BCB9BA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">864×120 19 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Here is the code:</p>
<pre data-code-wrap="python"><code class="lang-python">W = Linear(edge_feature_in, hidden_channels * 3) * C.view(-1, 1)
</code></pre>
<p>what does <code>pytorch</code> do in this bubble? Can I eliminate this?</p>
PyTorch3D: clarification on radius in FoVOrthographicCameras

username: Alex_Antony
cooked: <p>I am trying to set up three lines, each line oriented along the coordinate axis. The axes range from -128 to 128 containing 256 points. The camera is orthographic and am trying to render an image of 256 by 256 pixels. I set the radius to 1, expecting to see 256 circles in each coordinate axis.</p>
<p>The image is see is not what I expected. I have attached the images with radius=1. and radius = 0.05</p>
<p>What am I doing wrong ?</p>
<blockquote>
<p>Blockquote<br>
<span class="hashtag-raw">#Set</span> up Data<br>
num_points = 256<br>
lim = 128<br>
x_points = [[x, 0, 0] for x in np.linspace(-lim, lim, num_points)]<br>
y_points = [[0, y, 0] for y in np.linspace(-lim, lim, num_points)]<br>
z_points = [[0, 0, z] for z in np.linspace(-lim, lim, num_points)]<br>
lines = x_points + y_points + z_points<br>
colors = [[1.0, 0.0, 0.0, 1] for _ in range(num_points*3)]<br>
verts = torch.Tensor(lines).to(device)<br>
rgb = torch.Tensor(colors).to(device)<br>
point_cloud = Pointclouds(points=[verts], features=[rgb])</p>
</blockquote>
<p><span class="hashtag-raw">#Setup</span> Image Capture<br>
R, T = look_at_view_transform(dist=100, elev=0, azim=0)<br>
cameras = FoVOrthographicCameras(device=device, R=R, T=T, znear=0.01, max_y=128,min_y=-128,max_x=128,min_x=-128)<br>
raster_settings = PointsRasterizationSettings( image_size=256, radius = 1., points_per_pixel = 1)<br>
rasterizer = PointsRasterizer(cameras=cameras, raster_settings=raster_settings)<br>
renderer = PointsRenderer(rasterizer=rasterizer,compositor=AlphaCompositor())</p>
<p><span class="hashtag-raw">#Render</span> Image<br>
images = renderer(point_cloud)<br>
plt.figure(figsize=(10, 10))<br>
plt.imshow(images[0, …, :3].cpu().numpy())</p>
<blockquote>
<p>Blockquote</p>
</blockquote>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/0/303282535f8d8db7a31459c6ff63acab0df4b131.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/303282535f8d8db7a31459c6ff63acab0df4b131" title="radius_1_0_0_05.PNG"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/0/303282535f8d8db7a31459c6ff63acab0df4b131_2_689x255.jpeg" alt="radius_1_0_0_05.PNG" data-base62-sha1="6Sn8ijqnbEaCJF8Lb1A7EjOxFXb" width="689" height="255" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/0/303282535f8d8db7a31459c6ff63acab0df4b131_2_689x255.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/3/0/303282535f8d8db7a31459c6ff63acab0df4b131_2_1033x382.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/0/303282535f8d8db7a31459c6ff63acab0df4b131_2_1378x510.jpeg 2x" data-dominant-color="6C5A5A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">radius_1_0_0_05.PNG</span><span class="informations">2412×894 128 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Loss_objectness is Nan

username: GGlearning
cooked: <p>Hi folks,</p>
<p>While training a Faster RCNN for which the code is available <a href="https://github.com/DIUx-xView/xview3-reference/blob/main/reference/train_reference.ipynb" rel="noopener nofollow ugc">here</a>, I am facing following problem. I guess there is some problem with the input or the target. But what is the problem exactly, I am unable to understand that. Please give suggestions.</p>
<pre><code class="lang-auto">Loss is nan, stopping training
{'loss_classifier': tensor(0.0168, grad_fn=&lt;NllLossBackward0&gt;), 'loss_box_reg': tensor(0.0019, grad_fn=&lt;DivBackward0&gt;), 'loss_objectness': tensor(nan, grad_fn=&lt;BinaryCrossEntropyWithLogitsBackward0&gt;), 'loss_rpn_box_reg': tensor(0.3363, grad_fn=&lt;DivBackward0&gt;)}
An exception has occurred, use %tb to see the full traceback.
SystemExit: 1
</code></pre>
<p>Training images are normalised and the values lie between 0 and 1. However, Targets are in the form of Dictionary. Following are details about the sample image patch and the targets before feeding to the model. Patch’s histogram is attached as well.</p>
<p>targets keys dict_keys([‘boxes’, ‘labels’, ‘length_labels’, ‘scene_id’, ‘chip_id’, ‘image_id’, ‘area’, ‘iscrowd’])<br>
targets boxes tensor([[226., 681., 236., 691.],<br>
[495.,  19., 505.,  29.],<br>
[495.,  12., 505.,  22.],<br>
[704., 704., 714., 714.],<br>
[703., 407., 713., 417.],<br>
[345., 749., 355., 759.],<br>
[336., 700., 346., 710.],<br>
[766., 712., 776., 722.]])<br>
targets labels tensor([1, 3, 3, 1, 1, 1, 2, 1])<br>
targets scene_id 590dd08f71056cacv<br>
targets chip_id tensor(858)<br>
targets image_id tensor(32)<br>
targets area tensor([100., 100., 100., 100., 100., 100., 100., 100.])<br>
targets iscrowd tensor([0, 0, 0, 0, 0, 0, 0, 0])<br>
torch.Size([3, 800, 800])<br>
torch.float32<br>
tensor(0.) tensor(1.)</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/3/c30c9217cd05826a35843d39a7867a6955df231d.png" alt="img1_target" data-base62-sha1="rPu2r0WRLTawPOYMf1iBxwoCxeB" width="276" height="208"></p>

username: colinlaganier
cooked: <p>I am having the same issue running it on my M1 MacBook. Have you managed to solve this problem and were you running it an M1/M2 too?</p>
[Solved] Will change in dataset be reflected on dataloader automatically?

username: jdhao
cooked: <p>Hi, I contruct my own dataset according to the data loading tutorial and using the standard Dataloader provide by PyTorch. The code is like this,</p>
<pre><code class="lang-python">train_set = MyDataset(some_parameter...)
train_loader = Dataloader(dataset=train_set, other_setting...)

for batch_idx , data, target in enumerate(train_loader):
     #training processing
</code></pre>
<p>After several training epoch, I would like to regenerate my training dataset, I have written a custom member funtion <code>regenerate_sample()</code> for <code>MyDataset</code>. So I can just use <code>train_set.regenerate_sample()</code> to change the samples in <code>train_set</code>.</p>
<p>But what I am not sure about is will this change be reflected on <code>train_loader</code>, i.e., will <code>train_loader</code> now generate batch of samples from the new dataset samples instead of the old dataset? Or  do I have to manually construct a new <code>Dataloader</code> object in order to use the newly updated dataset?  like the following,</p>
<pre><code class="lang-auto">train_set.regenerate_sample()
train_loader = Dataloader(dataset=train_set, other_setting...)
</code></pre>
<p>Which is the case? I am a little confused.</p>

username: ptrblck
cooked: <p>The <code>DataLoader</code> seems to use a reference to the <code>Dataset</code> object, so that your regenerate should work.<br>
Here is a small sample code, which works fine for me:</p>
<pre><code>class MyDataset(Dataset):
    def __init__(self, data):
        self.data = data

    def __getitem__(self, index):
        return self.data[index]

    def __len__(self):
        return len(self.data)

    def regenerate(self):
        self.data = torch.ones(3, 1).float()


data = torch.zeros(3, 1).float()
dataset = MyDataset(data)
print dataset[0] # should output 0

data_loader = DataLoader(dataset)

for d in data_loader:
    print d

dataset.regenerate()
print dataset[0] # should output 1

for d in data_loader:
    print d</code></pre>

username: jdhao
cooked: <p>Yes, seems that a reference to <code>Dataset</code> is used. Thanks for your example~</p>

username: Rakshit_Kothari
cooked: <p>In case this helps anyone, I can confirm the switch successfully occurs for workers &gt; 0. This means that any modifications done to the <code>Dataset</code> object will traverse to the workers as well.</p>

username: KawaiiNotHawaii
cooked: <p>What if we are using multiple machines each equipped with multiple GPUs? Should we only do the update of the dataset on rank 0?</p>