How to divide the data into chunk

username: pramesh
cooked: <p>ROOT = ‘.data’</p>
<p>train_data = datasets.MNIST(root = ROOT,<br>
train = True,<br>
download = True)</p>
<p>mean = train_data.data.float().mean() / 255<br>
std = train_data.data.float().std() / 255</p>
<p>train_transforms = transforms.Compose([<br>
transforms.RandomRotation(5, fill=(0,)),<br>
transforms.RandomCrop(28, padding = 2),<br>
transforms.ToTensor(),<br>
transforms.Normalize(mean = [mean], std = [std])<br>
])</p>
<p>test_transforms = transforms.Compose([<br>
transforms.ToTensor(),<br>
transforms.Normalize(mean = [mean], std = [std])<br>
])<br>
VALID_RATIO = 0.9</p>
<p>n_train_examples = int(len(train_data) * VALID_RATIO)<br>
n_valid_examples = len(train_data) - n_train_examples<br>
train_data, valid_data = data.random_split(train_data,<br>
[n_train_examples, n_valid_examples])</p>
<p>i need to devide the train_data into 10 chunk …please support me…i am new to pytorch</p>

username: eqy
cooked: <p>If you want some folds for cross validation a simple thing to do is to shuffle the data once beforehand (perhaps deterministically by setting the seed manually to a fixed value beforehand).<br>
If you split data into 10 chunks, you can do something like:</p>
<pre><code class="lang-auto">data_len = len(train_data)
nchunks = 10
chunksiz = data_len/nchunks
chunks = list()
for i in range(nchunks):
    start_idx = i*nchunks
    chunks.append(train_data[start_idx:start_idx+chunksiz])
</code></pre>

username: pramesh
cooked: <pre><code class="lang-auto">data_len = len(train_data)
nchunks = 10
chunksiz = int(data_len/nchunks)
chunks = list()
for i in range(nchunks):
    start_idx = i*nchunks
    chunks.append(train_data[start_idx:start_idx+chunksiz])

i got the follwoing result..
ValueError: only one element tensors can be converted to Python scalars

</code></pre>
<p>can you suggest me how to resolve this issue?</p>
<p>Thank you so much for your respone…</p>

username: eqy
cooked: <p>Can you post the line that causes that issue? I can’t reproduce that on my end with</p>
<pre><code class="lang-auto">import torch

train_data = torch.rand(100, 32, 32)
data_len = len(train_data)
nchunks = 10
chunksiz = int(data_len/nchunks)
chunks = list()
for i in range(nchunks):
    start_idx = i*nchunks
    chunks.append(train_data[start_idx:start_idx+chunksiz])
print([chunk.shape for chunk in chunks])
</code></pre>
<pre><code class="lang-auto">[torch.Size([10, 32, 32]), torch.Size([10, 32, 32]), torch.Size([10, 32, 32]), torch.Size([10, 32, 32]), torch.Size([10, 32, 32]), torch.Size([10, 32, 32]), torch.Size([10, 32, 32]), torch.Size([10, 32, 32]), torch.Size([10, 32, 32]), torch.Size([10, 32, 32])]
</code></pre>

username: AlphaBetaGamma96
cooked: <p>If your <code>train_data</code> and <code>valid_data</code> are imported as <code>Tensors</code> you can use <code>torch.chunk</code> to split your Tensors into <code>n</code> chunks. The docs for <code>torch.chunk</code> is here → <a href="https://pytorch.org/docs/stable/generated/torch.chunk.html" class="inline-onebox" rel="noopener nofollow ugc">torch.chunk — PyTorch 1.9.0 documentation</a></p>

username: pramesh
cooked: <p>Yeah that is absolutly working fine.But while i am trying with mnist dataset…its not working…it showing error as i mentioned…</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import torch.utils.data as data

import torchvision.transforms as transforms
import torchvision.datasets as datasets


ROOT = '.data'

train_data = datasets.MNIST(root = ROOT, 
                            train = True, 
                            download = True)

mean = train_data.data.float().mean() / 255
std = train_data.data.float().std() / 255
train_transforms = transforms.Compose([
                            transforms.RandomRotation(5, fill=(0,)),
                            transforms.RandomCrop(28, padding = 2),
                            transforms.ToTensor(),
                            transforms.Normalize(mean = [mean], std = [std])
                                      ])

test_transforms = transforms.Compose([
                           transforms.ToTensor(),
                           transforms.Normalize(mean = [mean], std = [std])
                                     ])
train_data = datasets.MNIST(root = ROOT, 
                            train = True, 
                            download = True, 
                            transform = train_transforms)

test_data = datasets.MNIST(root = ROOT, 
                           train = False, 
                           download = True, 
                           transform = test_transforms)

VALID_RATIO = 0.9

n_train_examples = int(len(train_data) * VALID_RATIO)
n_valid_examples = len(train_data) - n_train_examples

train_data, valid_data = data.random_split(train_data, 
                                           [n_train_examples, n_valid_examples])

data_len = len(train_data)
nchunks = 10
chunksiz = int(data_len/nchunks)
chunks = list()
for i in range(nchunks):
    start_idx = i*nchunks
    chunks.append(train_data[start_idx:start_idx+chunksiz])
print([chunk.shape for chunk in chunks])
</code></pre>