akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Cafe Style Preprocessing in PyTorch for CUB_200

username: papoo13
cooked: <p>I am training a Bayesian neural network (based on this work) on CUB_200. My base architecture is this, a torchvision’s ResNet18. When training on this model my cross_entropy loss becomes nan in the first iteration. I am trying to debug it. While debugging, I noticed that my images are not normalized in a way that their values are between 0 and 1 or -1 and 1 but they have been normalized in a cafe style. This is  the code:<br>
For every image:</p>
<pre><code class="lang-auto">def _read_images_from_list(imagefile_list):
    imgs = []
    for imagefile in imagefile_list:
        #print("Reading img: ", imagefile)
        img = cv2.imread(imagefile).astype(np.float32)
        img = cv2.resize(img, (224, 224))
        # Convert RGB to BGR
        img_r, img_g, img_b = np.split(img, 3, axis=2)
        img = np.concatenate((img_b, img_g, img_r), axis=2)
        img -= np.array((103.94, 116.78, 123.68), dtype=np.float32)  # BGR mean
        #img -= np.array((123.68, 116.78, 103.94), dtype=np.float32) # RGB mean
        # HWC -&gt; CHW, compatible with pytorch
        img = np.transpose(img, [2, 0, 1])


        imgs += [img]
    return imgs
</code></pre>
<p>and then later in the dataloader’s <strong>getitem</strong>() function they convert images to torch:</p>
<pre><code class="lang-auto">    def __getitem__(self, index):
        img = torch.from_numpy(self._images[index])
        target = self._labels[index]
        task_labels = self.task_labels
        return img, target, task_labels
</code></pre>
<p>I wonder why they chose this type of normalization and why the channels are changed in this way? since the model is a torch model, I assume RGB should be fine?<br>
Thank you in advance!</p>
Pure C API in libtorch?

username: calandracas
cooked: <p>Hi I’m wondering if there is a C wrapper for the C++ API in libtorch?</p>
<p>I have read a few mentions of it, but haven’t come across any documentation or official mention of a C api, so maybe it doesn’t actually exist?</p>
<p>My project is in pure C and Fortran.</p>
<p>If one does not exist, I suppose my best bet would be to use the older Torch7?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="calandracas" data-post="1" data-topic="167181">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/73ab20/48.png" class="avatar"> calandracas:</div>
<blockquote>
<p>I’m wondering if there is a C wrapper for the C++ API in libtorch?</p>
</blockquote>
</aside>
<p>Im not aware of such a wrapper.</p>
<aside class="quote no-group" data-username="calandracas" data-post="1" data-topic="167181">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/73ab20/48.png" class="avatar"> calandracas:</div>
<blockquote>
<p>I suppose my best bet would be to use the older Torch7?</p>
</blockquote>
</aside>
<p>I would also not recommend trying to use the deprecated Torch7 releases, as they wouldn’t even work with the latest GPU families. Would it be possible to use the C++ libtorch API instead?</p>

username: calandracas
cooked: <p>Thanks for the information.</p>
<p>I’m new to machine learning, and I think that PyTorch probably isnt the right tool for the job.</p>
<p>I’m trying to add a neutral network to my application for edge devices.</p>
<p>Using C++ may be possible, but I have a personal preference for C</p>
<p>It seems to me that something like tensorflow lite is likely to be the type of tool im looking for.</p>
Objectness and box regression in RPN - backpropagation

username: sigma_x
cooked: <p>There are two steps that confuse me in rpn.py:</p>
<pre><code class="lang-auto">252                 objectness = objectness.detach()
370                 proposals = self.box_coder.decode(pred_bbox_deltas.detach(), anchors)
</code></pre>
<p>So gradients are detached in both class and box predictions. I don’t understand then, how loss is backpropagated through RPN, because it is computed in RPN and RoI after the detached tensors?</p>
Assigning weights to individual datasets in ConcatDataset

username: Laya1
cooked: <p>I have three datasets with <code>1600</code>, <code>400</code> and <code>200</code> images respectively. I use <code>ConcatDataset</code> to merge the images in the three datasets. However, I see that the model learns to perform well on the first dataset (with <code>1600</code> images) and performs poorly on the other two. Is it possible to assign weights to the three datasets so that the network sees the images from second and third datasets more often that the images from first dataset. Specifically, I want to give weights to the individual datasets as <code>2200/1600</code>, <code>2200/400</code> and <code>2200/200</code> so that the images are sampled as I want. Also, is there any other way to achieve this? Thanks in advance for any help.</p>

username: ptrblck
cooked: <p>You could use a <code>WeightedRandomSampler</code> as described <a href="https://discuss.pytorch.org/t/using-weightedrandomsampler-with-concatdataset/51968/2">in this post</a>.</p>

username: Laya1
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thank you for your reply. I do not need the targets since I am training for object detection. So, I am using the following method:</p>
<pre><code class="lang-auto">dataset_train = torch.utils.data.ConcatDataset([dataset_train_1, dataset_train_2, dataset_train_3])

weights_train = [
    [dataset_train.__len__() / dataset_train_1.__len__()] * dataset_train_1.__len__(),
    [dataset_train.__len__() / dataset_train_2.__len__()] * dataset_train_2.__len__(),
    [dataset_train.__len__() / dataset_train_3.__len__()] * dataset_train_3.__len__(),
]
weights_train = list(itertools.chain.from_iterable(weights_train))
sampler_train = torch.utils.data.WeightedRandomSampler(weights=weights_train, num_samples=len(weights_train))
</code></pre>
<p>This is similar to what is done in the post, except I am not weighing according to the targets, but rather the source dataset. Thank you again for your suggestion.</p>
Equality constrained optimization

username: WowPy
cooked: <p>Hello,</p>
<p>I am interested in performing equality-constrained optimization using PyTorch. In other words, let’s say that I have p, q as inputs to the NN, where Z is the output. The examples are split into two types: 1) the mapping from p,q to Z cannot be violated. So, I need the network to return the exact target values. 2) For the rest of the examples (p, q), Z is found using the regression model to be developed, and it is okay to be a bit off from the targets.</p>
<p>So, I am interested in doing this equality-constrained problem; any insights/feedback is much appreciated.</p>
How to collect whole batch at once?

username: malfonsoarquimea
cooked: <p>Hi!<br>
I am now working on a problem where I must read my data from a table-like h5 file<br>
The problem is that, given how I am providing the sampling (with a batch_sampler) and how the h5 file is structured, the time to read one item is almost the same as the time to read the whole batch (25k items).<br>
I would like to know if there is some way to tell the dataset and dataloader to load one whole batch at a time to avoid reading multiple time from the hdf5</p>

username: ptrblck
cooked: <p>You could use a <code>BatchSampler</code> and pass all indices directly to <code>Dataset.__getitem__</code> which would allow you to load multiple samples. <a href="https://discuss.pytorch.org/t/how-to-use-batchsampler-with-getitem-dataset/78788/4">This code</a> shares an example.</p>

username: malfonsoarquimea
cooked: <p>I ended up doing exactly that, thanks!</p>
Different weight initilization for each dense block, densenet121

username: haj2022
cooked: <p>[I can not seem to find a way to initialize weights for different dense blocks.<br>
I am using <strong>for name,layer in model.named_modules()</strong> for accessing layers.I initialize weights of ‘feature.denseblock2’ by a custom weight and rest by Xavier initialization.<br>
the issue is the layers in each dense block goes to Xavier initialization as the it loops through named_modules.<br>
return_node.keys() = ‘feature.denseblock2’</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/b/2babd07917dae7556fb975433885be1ec3b3cc76.png" alt="Screenshot from 2022-12-02 13-02-11" data-base62-sha1="6ekDMY99ZQb0b4LDCwUZkwtCjFs" width="526" height="434"></p>
Different top1 run multiple times

username: Liangzhi_Tang
cooked: <p>torch 1.11.0+cu113, mobilenet_v2  pretrained weights<br>
Run three times, it gives three top1:<br>
71.72, 71.84, 71.75</p>
<p>Why?</p>

username: huahuanZ
cooked: <p>Have you invoked <code>model.eval()</code> before evaluation.</p>
Two forward with same network to get two loss, why two backward works without retain_graph=True?

username: zai_heshi
cooked: <pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.optim as optim

class Linear(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(4, 3)

    def forward(self, x):
        output = self.linear(x)
        return output
</code></pre>
<pre><code class="lang-auto">def p1():
    linear_layer = Linear()
    optimizer = optim.SGD(linear_layer.parameters(), lr=0.001)

    x0 = torch.randn((3, 4))
    x1 = torch.randn((3, 4))

    output_1 = linear_layer(x0)
    output_2 = linear_layer(x1)
    loss1 = output_1.mean()
    loss2 = output_2.mean()
    optimizer.zero_grad()
    loss1.backward()
    loss2.backward()  # why this works?
    optimizer.step()
    for name, params in linear_layer.named_parameters():
        print('name: ', name)
        print('grad: ', params.grad)
</code></pre>
<pre><code class="lang-auto">def p2():
    linear_layer = Linear()
    optimizer = optim.SGD(linear_layer.parameters(), lr=0.001)

    x = torch.randn((3, 4))

    output_1 = linear_layer(x)
    loss1 = output_1.mean()
    loss2 = output_1.sum()
    optimizer.zero_grad()
    loss1.backward(retain_graph=True) # why this need set retain_graph=True
    loss2.backward()
    optimizer.step()
    for name, params in linear_layer.named_parameters():
        print('name: ', name)
        print('grad: ', params.grad)
</code></pre>
<p>Hello,  I know the computational graph will be freed after calling <code>backward()</code>. Can anyone explain why p1 not need set retain_graph=True? Is it because two computational graphs are constructed?</p>

username: huahuanZ
cooked: <p><code>retain_graph=True</code> is required only if you conduct backward through non-leaf tensor multiple times.</p>
<p>In the <code>p1()</code>, backward goes through the <code>linear_layer.weight/bias</code> twice, but they are leaf tensors.<br>
In the <code>p2()</code>, <code>output_1</code> is a non-leaf tensor created by <code>linear_layer</code>.</p>
How to choose the parameter in the loss function?

username: msr3384
cooked: <p>Hello everyone,</p>
<p>I have the loss function:</p>
<p>1/2 * torch.norm(output.mT - output.mT@weight, ‘fro’)**2 + gamma * torch.sum(abs(weight)) + lamda/4*torch.trace((torch.square(output@output.mT - torch.eye(len(output)))))</p>
<p>In the above loss function there are 3 components and obviously the have values of different sizes. Now if I choose my parameter gamma and lambda any random numbers then the during the training the output of the model collapse to zero ( I posted earlier about this). Now if I choose lambda and gamma such that second part and the last part of the loss function become smaller after multiplied by lambda and gamma then the output does not collapse to zero anymore and the loss function also decreases. Now my concern I choose the parameter once I observe the values of the second and the last components of the loss function but what is the general technique to chose the parameters involves in the loss function so that loss function works?</p>
Tricky matrix multiplication

username: Konstantin_Suloev_Jr
cooked: <p>I am computing attention weights and i want to make it vectorized.<br>
I have the following tensors:<br>
input tensor is [batch_size, channels, x, y], weight tensor is [channels, channels] (to match the dimensions)<br>
for each x and y and for each image i need to multiply a vector  [channels] by weight tensor.<br>
in my terms the shapes are [64, 256, 25, 2] and [256, 256]. I am willing to see output like [64, 256, 50].<br>
Is it possible to do so using basic operations?</p>

username: AlphaBetaGamma96
cooked: <p>Hi <a class="mention" href="/u/konstantin_suloev_jr">@Konstantin_Suloev_Jr</a>,</p>
<p>The following works, but check if the reshape function makes sense for your particular use-case.</p>
<pre><code class="lang-auto">x=torch.randn(64,256,25,2)
w=torch.randn(256,256)

out = torch.einsum("bcx,cd-&gt;bdx",x.reshape(64,256,50), w)
print(out.shape) #returns torch.Size([64, 256, 50])
</code></pre>

username: Konstantin_Suloev_Jr
cooked: <p>Thank you for your answer! I bet i got another solution using nn module</p>
<p>Ws = nn.Linear(256, 256)<br>
result = Ws(input.flatten(2).permute([0, 2, 1]))<br>
it seems that it makes the same thing as you suggested</p>

username: AlphaBetaGamma96
cooked: <p>If <code>Ws</code> has its bias set to False, then yes. If not, then they’re different operations.</p>
<p>They’ll be equivalent (with <code>bias = False</code>), but I just merged the operations into a single einsum string.</p>
Recommended way to build datapipe grouped/batched by csv file

username: thomasgho
cooked: <p>Trying to load protein data from many csv files. Each csv file corresponds to one protein and contains multiple rows (the number of rows varies for each csv file). I’d like to build a datapipe such that the protein data is grouped (like having uneven batches, with each batch representing one protein).</p>
<p>The default csv parser and the examples online yield one row at a time. Currently, to group the individual rows to their corresponding protein, I use the default csv parser (yielding one row at a time) with <code>return_path=True</code>. Then I use <code>Grouper</code> to group rows with the same file path:</p>
<pre><code class="lang-auto">def build_datapipe(root_dir):
    datapipe = dp.iter.FileLister(root_dir, recursive=True)
    datapipe = dp.iter.FileOpener(datapipe, mode='rt')
    datapipe = datapipe.parse_csv(skip_lines=1, return_path=True)
    datapipe = datapipe.map(preprocess) # row-wise preprocessing
    datapipe = datapipe.groupby(group_key_fn=group_fn)
    return datapipe
</code></pre>
<p>Is there a recommended way to do this? Perhaps writing a custom csv parser that yields multiple rows at a time.</p>

username: nivek
cooked: <p>I think writing a custom DataPipe to yield a list of row from the same CSV makes sense. Or you can write a custom sequential grouper that is similar to <a href="https://pytorch.org/data/main/generated/torchdata.datapipes.iter.ParagraphAggregator.html" rel="noopener nofollow ugc">ParagraphAggregator</a>.</p>
How do my transforms affect my model?

username: gifted_spence
cooked: <p>Hey,</p>
<p>I don’t understand why my transformations lead to my model failing to run (RuntimeError: mat1 and mat2 shapes cannot be multiplied (4x44944 and 400x120)). Generally, I stick to the documentation of training a classifier (<a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Training a Classifier — PyTorch Tutorials 1.13.0+cu117 documentation</a>) only that I transformed my data differently.</p>
<pre><code class="lang-auto">transform = transforms.Compose([transforms.RandomRotation(30),
                                transforms.RandomResizedCrop(224),
                                transforms.RandomHorizontalFlip(),
                                transforms.ToTensor(),
                                transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])
</code></pre>
<p>Looking at the shape of my batch tensor I don’t see why the model as indicated in the documentation now doesn’t run anymore. Do these transformation change something that is relevant to the model?</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = torch.flatten(x, 1) # flatten all dimensions except batch
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x
</code></pre>
<p>In general, I know how to fix this error, but I don’t understand what makes my tensor so big all of a sudden. Since nothing is changed compared to the documentation it must have something to do with my previous transforms. Can someone explain, what happened?</p>
<p>I would be happy about any help!<br>
Cheers</p>

username: ptrblck
cooked: <p>The tutorial uses <code>CIFAR10</code> where samples have a spatial size of <code>32x32</code> while you are resizing your samples to <code>224x224</code> and are thus increasing the number of features in the flattened activation.</p>
Training tricks to improve stability of mixed precision

username: Kinbote
cooked: <p>I would love to be able to use automatic mixed precision more extensively in my training, but I find that it is too unstable and often ends in NaNs. Are there any general tricks in training that people here have used to improve stability?</p>
<p>I’ve seen the following general tips:</p>
<ul>
<li>plot the gradients and force unstable layers to fp32</li>
<li>bump weight decay in the optimizer</li>
<li>bump epsilon in the optimizer</li>
<li>try an exotic optimizer</li>
<li>add/try different normalization layers</li>
<li>force loss calculations to fp32</li>
</ul>

username: ptrblck
cooked: <p>The mixed-precision training utils. via <code>torch.amp</code> should already cast to the appropriate <code>dtype</code> if the corresponding layer would otherwise suffer from the decreased numerical stability (i.e. in particular via the <code>torch.cuda.amp.autocast</code> context).<br>
I’ve seen that normalization layers were generally beneficial: plain FP32 training “exploded” while mixed-precision training ran into overflows and couldn’t recover.</p>
Cannot make LSTM deterministic

username: pozimekSG
cooked: <p>The <a href="https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html#torch.nn.LSTM" rel="noopener nofollow ugc">LSTM documentation</a> suggests that it’s possible to make the LSTM deterministic if the right environment variable is set. I found that not to be the case. Minimal reproduction is included below. If this is intended behavior then the documentation should be updated so as not to mislead users.</p>
<p>lstm_test.py:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

torch.use_deterministic_algorithms(True)
seed = 1
model= nn.LSTM(4, 4, 2, batch_first = True, dropout=0.5, bidirectional=True).cuda()

for i in range(3):
    torch.manual_seed(seed)
    torch.cuda.manual_seed(seed)
    model.train()
    data = torch.randn(4, 4)
    if i &gt; 0:
        print(torch.equal(data, pre_data))
    pre_data = data
    data = data.cuda()
    out, _ = model(data)
    loss = out.sum()
    print(i, loss.item())
</code></pre>
<p>Calling in bash results in the following:</p>
<pre><code class="lang-auto">&gt; CUBLAS_WORKSPACE_CONFIG=:4096:2 python lstm_test.py
0 -3.0934510231018066
True
1 -2.6078720092773438
True
2 -2.4402637481689453
</code></pre>
<p>My env:<br>
Ubuntu 18.04.6 LTS<br>
torch.<strong>version</strong> = 1.12.1<br>
torch.version.cuda = 11.3<br>
sys.version = 3.9.13</p>
Accessing one element from data_loader

username: sroyyors
cooked: <p>hi,<br>
I would like to get just one batch from the data_loader and push it through the model. I use the MNIST cpp example to make the data loader as:</p>
<p>auto train_loader = torch::data::make_data_loader<a>torch::data::samplers::SequentialSampler</a>(std::move(train_dataset), batch_size);</p>
<p>But how do I access say the first or tenth batch? I don’t want to use the for loop iterator.</p>
<p>I tried train_loader.get(1) or train_loader.next() but they don’t work.</p>

username: saluei
cooked: <p>this may help you, just return 10 first sample from dataset</p>
<pre><code class="lang-auto">trainset = torchvision.datasets.MNIST(root=DtSetRootPath, train=True,download=True)
trainset.data=trainset.data[0:10,:,:,:]
trainloader = torch.utils.data.DataLoader(trainset, batch_size=10,shuffle=True,num_workers=0)

testset = torchvision.datasets.MNIST(root=DtSetRootPath, train=False,download=True)
testset.data=testset.data[0:10,]
testloader = torch.utils.data.DataLoader(testset, batch_size=10,shuffle=True, num_workers=0)

return trainloader,testloader
</code></pre>

username: sroyyors
cooked: <p>Thank you so much! This is very helpful. I also found a way to access trainload as</p>
<blockquote>
<p>auto dat=train_loader-&gt;begin()<br>
auto&amp; eg2=dat-&gt;data;</p>
</blockquote>
<p>I seem to have not interpreted train_loader’s type properly.<br>
thanks,<br>
Sushmita</p>
Conv2d with 5D tensor

username: Vilma_Hagg_Edelonn
cooked: <p>Hello,</p>
<p>I have a list of frames (X_train) represented as a tensor, it has the shape of (5,150,200), so 5 frames of size 150x200.</p>
<p>I want to use X_train in a conv2d layer which I believe takes in a 3D (unbatched) or 4D (batched) tensor. My tensor is 5D (batch,channel,dim,height,width) because of it being multiple frames. I tried making it (batch,channel,height,(dim*width)), but then I get the error: ValueError: Expected input batch_size (1) to match target batch_size (5).</p>
<p>How could I solve this? I am very new to pytorch and despite reading documentation I have a hard time figuring it out.</p>
<p>Any help is appreciated!</p>

username: KFrank
cooked: <p>Hi Vilma!</p>
<aside class="quote no-group" data-username="Vilma_Hagg_Edelonn" data-post="1" data-topic="167284" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/v/b3f665/48.png" class="avatar"> Vilma_Hagg_Edelonn:</div>
<blockquote>
<p>I want to use X_train in a conv2d layer which I believe takes in a 3D (unbatched) or 4D (batched) tensor. My tensor is 5D (batch,channel,dim,height,width) because of it being multiple frames.</p>
</blockquote>
</aside>
<p>Speculating as to the details of your use case, I assume that you want to<br>
apply your <code>Conv2d</code> to each of the five frames independently, that is, you<br>
don’t want the five frames to mix together.  (Let me further assume that<br>
you <em>do</em> want the channels to mix together as they normally would when a<br>
<code>Conv2d</code> is applied to a <code>[batch, channel, height, width]</code> 4d tensor.)</p>
<p>If this is the case, you would use <code>transpose()</code> to bring the <code>dim</code> (frames)<br>
and <code>batch</code> dimensions together, and then use <code>reshape()</code> to merge those<br>
two dimensions into a single “batch” dimension:</p>
<pre data-code-wrap="python"><code class="lang-python">input4d = input5d.transpose (1, 2).reshape (batch * dim, channel, height, width)
result4d = conv2d (input2d)
</code></pre>
<p>(You could then, if appropriate, convert <code>result4d</code> back into your 5d format<br>
by calling <code>reshape()</code> and <code>transpose()</code> to undo the original <code>transpose()</code><br>
and <code>reshape()</code>.)</p>
<p>Best.</p>
<p>K. Frank</p>
Self.log doesn't work in PyTorch Lightning

username: ali_hejrati
cooked: <p>I create a custom Logger module like the one below, and when I call self.log in training_step nothing happens. only MyLogger.save method is called and I don’t get metrics</p>
<pre><code class="lang-auto">class MyLogger(Logger):
    def __init__(
        self, 
        save_dir: str = None,
        name: Optional[str] = 'GeineLogger',
        fn_name: Optional[str] = '_genie',
        version: str = '0.1',
        **kwargs: Any
    ):
        super().__init__(**kwargs)
        self._save_dir = save_dir
        self._name = name
        self._fn_name = fn_name
        self._version = version

    @property
    def name(self):
        return self._name
    
    @property
    def fn_name(self):
        return self._fn_name

    @property
    def version(self):
        # Return the experiment version, int or str.
        return self._version

    @property
    def save_dir(self) -&gt; Optional[str]:
        """Return the root directory where experiment logs get saved, or `None` if the logger does not save data locally."""
        return self._save_dir

    @rank_zero_only
    def log_hyperparams(self, params):
        # params is an argparse.Namespace
        # your code to record hyperparameters goes here
        logger.critical('log_hyperparams | params={}'.format(params))
        pass

    @rank_zero_only
    def log_metrics(self, metrics, step):
        # metrics is a dictionary of metric names and values
        # your code to record metrics goes here
        logger.critical('log_hyperparams | metrics={}'.format(metrics))
        pass

    @rank_zero_only
    def save(self):
        # Optional. Any code necessary to save logger data goes here
        super().save()
        logger.critical('save')
        pass

    @rank_zero_only
    def finalize(self, status):
        # Optional. Any code that needs to be run after training
        # finishes goes here
        super().finalize(status)
        logger.critical('finalize | status={}'.format(status))
        pass
</code></pre>
Keep seed constant for dataloader/transforms but change seed for weight initialization

username: JuliaWasala
cooked: <p>Hi, I’m training a CNN on GPU with a custom dataset + transforms. I need it to be fully reproducible, so now I have set the random seed with torch.manual_seed(seed). I want to train+eval the CNN 10 times, with different weight initializations. As far as I understand, I can control these weight initializations with the random seed. However, I want to keep the seed for the dataloader/transforms constant.</p>
<p>Questions:</p>
<ul>
<li>Is it possible to change the seed temporarily only for model initialization, and then change it back to the “default” I use for the transforms?</li>
<li>Where exactly do I need to set the seed? My code is divided over multiple files/functions: a class for the CNN, a class for the data, function for the dataloaders. Training is divided into 3 functions (1 epoch, a general training function and a specific train + eval function for the current model I’m training), and finally the top-level script that runs the training+eval.</li>
</ul>
<p>torch version 1.13, torchvision 0.14.0, python 3.9.13</p>
Parquet file reader cannov vonerte tuple to pyarrow.lib.NativeFile

username: nleh
cooked: <p>Hi,</p>
<p>I want to use TorchData and DataPipes for a large parquet timeseries file with several different timeseries. I can open and work with the parquet file in polars etc, but I cannot manage with torchdata.</p>
<pre><code class="lang-auto">dp = FileLister(root=data_dir).filter(lambda fname: fname.endswith(".parquet"))
dp = FileOpener(dp)
parquet_dp = dp.load_parquet_as_df()
stream = parquet_dp.batch(batch_size=16)
print(list(stream)[0])
</code></pre>
<p>yields</p>
<pre><code class="lang-auto">TypeError: Cannot convert tuple to pyarrow.lib.NativeFile
This exception is thrown by __iter__ of ParquetDFLoaderIterDataPipe(columns=None, device='', dtype=None, source_dp=FileOpenerIterDataPipe, use_threads=False)
</code></pre>
<p>I am using the following:</p>
<pre><code class="lang-auto">Ubuntu 20.04
python 3.9.13
torchdata 0.5.0
torcharrow 0.2.0a0.dev20221129
pyarrow 0.8.0
polars 0.14.28
</code></pre>

username: nivek
cooked: <p>Is your Parquet file an Arrow table or similar? Would it be readable with <a href="https://arrow.apache.org/docs/python/generated/pyarrow.parquet.ParquetFile.html" rel="noopener nofollow ugc"><code>pyarrow.parquet.ParquetFile</code></a>?</p>
<p>If not, you may have to write your own custom DataPipe to parse it.</p>

username: nleh
cooked: <p>I created the parquet file from a polars dataframe.</p>
<p>The following both run without error:</p>
<pre><code class="lang-auto">table = pq.read_table(data_dir)
parquet_file = pq.ParquetFile(data_dir)
</code></pre>
<p>Edit:<br>
I believe I found the issue, but don’t know why it occurs. In the <code>torchdata/datapipes/iter/util/dataframemaker.py</code> source code file, line 144</p>
<pre><code class="lang-auto">for path in self.source_dp:
     parquet_file = parquet.ParquetFile(path)
</code></pre>
<p>path here is a tuple of <code>(data_dir, StreamWrapper&lt;&lt;_io.TextIOWrapper name=data_dir mode='r' encoding='UTF-8'&gt;&gt;)</code> and hence the error gets thrown inside pyarrow.</p>

username: nleh
cooked: <p>I found the error, one does not need to open the file before using the parquet file reader, so the line <code>dp = FileOpener(dp)</code> has to be removed.</p>