How to use fast and slow workers together in DataLoader?

Question:
Hello everyone, I’m a newbie to pytorch.
There are two data simulation approaches in my training, one works fast, and one works much slower. Worker id is used to distinguish the two approaches, e.g. there are 10 workers in total, worker 0 to 8 use fast simulation, and worker 9 uses slow simulation.
Since multi-processes are used in DataLoader, it is supposed that the DataLoader and the training process works like the producer-consumer mode: once a data batch is produced by a worker, it is add to a queue. On the other side, the training process get data batches from the queue, and wait if the queue is empty.
However, it is found that the training time is the same as all workers use the slow approach. So, I deduce that workers are not owned by independent subprocesses, but run in a loop, the data simulation is slowed down by the worker 9.
My DataLoader is initialized like this:
dataloader = torch.utils.data.DataLoader(dataset, batch_size = BATCH_SIZE, num_workers = NUM_WORKERS)
My question is that is there are any way to make the process not stucked by the slow workers? Is there are any parameters I missing? Thank you very much!

Context:
For example, I have tried this toy dataset:
import numpy as np
import torch
import time

class ToyDataset(torch.utils.data.IterableDataset):
    def __init__(self, numworkers):
        super().__init__()
        self.numworkers = numworkers
        self.batchidx = 0

    def __iter__(self):
        return self

    def __next__(self):
        id = 0
        info = torch.utils.data.get_worker_info()
        if info is not None:
            id = info.id

        if id < self.numworkers - 1:
            time.sleep(0.5)
            print('fast worker {:d}, batch {:d}'.format(id, self.batchidx))
        else:
            time.sleep(5)
            print('-slow worker {:d}, batch {:d}'.format(id, self.batchidx))

        retval = self.batchidx
        self.batchidx += 1
        return retval

BATCH_SIZE = 1
NUM_WORKERS = 2
TIMEOUT = 0
PREFETCH_FACTOR = 2

dataset = ToyDataset(NUM_WORKERS)
dataloader = torch.utils.data.DataLoader(
    dataset,
    batch_size = BATCH_SIZE,
    num_workers = NUM_WORKERS,
    timeout = TIMEOUT,
    prefetch_factor = PREFETCH_FACTOR)
it = iter(dataloader)

for i in range(1000):
    try:
        data = next(it)
        print('----------')
    except RuntimeError:
        print('timeout')

The output is something like this:
fast worker 0, batch 5
-slow worker 1, batch 3

fast worker 0, batch 6
-slow worker 1, batch 4

fast worker 0, batch 7
-slow worker 1, batch 5

The data generation is always slowed down by the slow worker.


Question:
Hey everyone,
I am still a PyTorch noob. I want to do Incremental Learning and want to split my training dataset (Cifar-10) into 10 equal parts (or 5, 12, 20, …), each part with the same target distribution.
I already tried to do it with sklearn (train_test_split) but it only can split the data in half:
from sklearn.model_selection import train_test_split

transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])
trainset = torchvision.datasets.CIFAR10(root='./data', train=True, download=True, transform=transform)

data1_idx, data2_idx= train_test_split(
np.arange(len(targets)),
test_size=0.5,
shuffle=True,
stratify=targets)

data1_sampler = torch.utils.data.SubsetRandomSampler(data1_idx)
data2_sampler = torch.utils.data.SubsetRandomSampler(data2_idx)

data1_loader = torch.utils.data.DataLoader(trainset, batch_size=4, sampler=data1_sampler)
data2_loader = torch.utils.data.DataLoader(trainset, batch_size=4, sampler=data2_sampler)

How would you do it in PyTorch? Maybe you can point me to some example code.

Context: I think `sklearn.model_selection.StratifiedKFold 1 might be useful as it allows you to create multiple splits in a stratified fashion.


Question:
Hi Pytorch community,
I am training a model on a very wide dataset (~500,000 features). To read the data from disc I use dask to load an xarray.core.dataarray.DataArray object to not load all the data in memory at once. I can load subsets of the data into memory with a numpy array as such: xarray[0:64,:].values. This loads 64 samples into memory in about 2 seconds. I then want to feed the data to the model one batch at a time, using batch size of 64, since this should have an estimated epoch time of 3 minutes given the total sample size (~6000). My problem is when I try to implement this functionality in the pytorch Datasetmodule. I want to create a class in which a DataLoader loads the data from disc into memory one batch at a time with a specified batch size. To that I made the following Datasetclass and wrapped it in a DataLoader.

class load_wide(Dataset):

    def __init__(self, xarray, labels):

        self.xarray = xarray
        self.labels = labels

    def __getitem__(self, item):

        data = self.xarray[item,:].values
        labels = self.labels[item]

        return data, labels

    def __len__(self):
        return len(self.xarray)

I then load the data like this:
train_ds = load_plink(xarray_train, labels_train)
train_dl = DataLoader(train_ds, batch_size=64, shuffle=True)

for i, data in enumerate(train_dl):
    feats, labels = data
    preds = net(feats)
    loss = criterion(preds, labels)

This works fine, but it takes about 90 seconds to load a batch resulting in unreasonable training time. What went wrong in my implementation of dataloading, that causes it to load the data so slowly. Using the DataLoader causes a 45-fold slowdown of dataloading. Can someone explain the cause of this?
Also, how can you simultaneously evaluate the model on a validation set that is loaded one batch at a time?

Context: I’m not familiar with the internal implementation of the xarray, but what seems to be different is the shuffling. Could you test your simple code snippet with random indices xarray[0:64,:].values instead of contiguous ones and compare the loading speed?


Help! adding data of the target subject to training performs worse

Question:
Hi Pytorch community,
I have a held-one-patient-out experiment. I have designed a CNN+RNN net. During training, since I have an imbalanced dataset I used the sampler within the DataLoader.
Now I am adding some labeled samples (e.g 10) to my training set from the target subject with the objective of improving the performance of my task. Although I would expect this to be beneficial for my model, in some situations I see the performance drops down. So my question is, how can I be sure the DataLoader is using also the added target samples? I would expect adding samples for the target subject would be at least as good as not adding them, but never worse. Any help would be more than appreciated.
I train my model from scratch. I stop the training at 20 epochs. I use BatchNorm with the PyTorch momentum param equal to 0.01
This is a sniped of my code.
import torch
torch.manual_seed(0) # the same seed, to ensure the same weight init

train_df = pd.DataFrame()
train_df = pd.concat([train_df, train_df_aux, seeds])
# train_df_aux is the training data without any labelled samples from the target subject
# seeds are the labelled samples from the target subject, this is also an umbalaced subset, since I get the labelled samples untill X positive class are found.
train_df.reset_index(drop=True, inplace=True)

# DATA LOADERS
train_data = torch.utils.data.ConcatDataset([train_data_ori, train_data_trf1])
# train_data_trf1 is the train data with augmentation strategies.

sampler = torch.utils.data.sampler.WeightedRandomSampler(weights, len(weights))
#weights is constructed using cutomized fuctions to have during training balanced batched.

# During training
kwargs = {'num_workers': hparams["num_workers"], 'pin_memory': True} if use_cuda else {}
train_loader = DataLoader(train_data, batch_size=hparams["batch_size"],
                          sampler=sampler,
                              **kwargs)

Thank you in advance!

Context: If you just iterate through DataLoader and print out each sample, do you see the label samples that you added?


Loading same data but getting different results

Question:
So I’m trying to manually split my training data into batches such that I can easily access them via indexing, and not relying on DataLoader to split them up for me, since that way I won’t be able to access the individual batches by indexing. So I tried the following:
train_data = datasets.ANY(root='data', transform=T_train, download=True)
BS = 200
num_batches = len(train_data) // BS
sequence = list(range(len(train_data)))
np.random.shuffle(sequence)  # To shuffle the training data
subsets = [Subset(train_data, sequence[i * BS: (i + 1) * BS]) for i in range(num_batches)]
train_loader = [DataLoader(sub, batch_size=BS) for sub in subsets]  # Create multiple batches, each with BS number of samples

Which works during training just fine.
However, when I attempted another way to manually split the training data I got different end results, even with all the same parameters and the following settings:
device = torch.device('cuda')
torch.manual_seed(0)
np.random.seed(0)
torch.backends.cudnn.benchmark = False
torch.backends.cudnn.deterministic = True
torch.cuda.empty_cache()

I only split the training data the following way this time:
train_data = list(datasets.ANY(root='data', transform=T_train, download=True))  # Cast into a list
BS = 200
num_batches = len(train_data) // BS
np.random.shuffle(train_data)  # To shuffle the training data
train_loader = [DataLoader(train_data[i*BS: (i+1)*BS], batch_size=BS) for i in range(num_batches)]

But this gives me different results than the first approach, even though (I believe ) that both approaches are identical in manually splitting the training data into batches. I even tried not shuffling at all and loading the data just as it is, but I still got different results (85.2% v.s 81.98% accuracy). I even manually checked that the loaded images from the batches match; and are the same using both methods.
Not only that, when I load the training data the conventional way as follows:
BS = 200
train_loader = DataLoader(train_data, batch_size=BS, shuffle=True)

I get even more drastic results!
Can somebody please explain to me why these differences arise, and how to fix it?

Context: You have confirmed that “the loaded images from the batches match” between the first two methods. What happens afterward? Do you set your seed (e.g. torch.manual_seed(0)) before training?


Doubt about cross-validation with training, validation and test set

Question:
hi all,
I have a question where I am apparently not able to find any answer.
Let’s say I have a dataset which is relatively small and I want to be able to test on the entire dataset to reduce the bias of my model.
I thought I could use something like k-fold cross-validation, but no matter wher I look, i only find the case where for each fold, the data is split in only training and test set. What I am interested in is to have for each fold a training, validation AND test data, where I train the model on the test data, determine when to stop training on the validation data and then test on the test data. I would then report the average of the scores obtained on the different test sets.
However, I am unsure on how to split the data and if the mentioned procedure is actually correct.
Would this for example be correct? (I fix [p] to be the validation set) and test on the other different test sets)
Dataset = [o, d, p, x]
Fold1: Train = [o, d], Validation = [p], Test=[x]
Fold2: Train = [o, x], Validation = [p], Test=[d]
Fold3: Train = [d, x], Validation = [p], Test=[o]
Thank for help!

Context:
@rasbt published this great post 4 a while ago where he explains cross validation in detail (you should also check other resources from his blog).
To perform the actual splitting you could use e.g. scikit-learn’s methods.


Training Mask R-CNN on MS COCO (single class): urllib error

Question:
The dataset interface runs problem-free for a couple of epochs, then outputs this error:

urllib.error.HTTPError: HTTP Error 503: Service Unavailable

when running the line with scikit-image method for opening a URL:

im = np.array(io.imread(self.img_data[idx][‘coco_url’]))

I’m training a single class (‘toothbrush’), and the model has 2 labels, background+class. Toothbrush has label 80, so I change 80 to 1 when training. None of the errors I could relate to are reported, just this one.
EDIT: another error that pops up (same method):

ConnectionResetError: [Errno 104] Connection reset by peer

Context:
I think it is possible that the server that the URLs are pointing have a quota or is overloaded, such that your request gets an HTTP Error 503.
Is it possible for you to download the data in advance and load them from disk?


How to use the size of the input of GNN layers for in_channels?

Question:
I am trying to use TransformerConv to train a model which takes in graphs of different sizes. This is to be used on a graph classification task. My code snippet is as follows:
self.conv1 = TransformerConv(in_channels=-1, out_channels=256)

def forward(self, x_onein, x_twoin, lambd):
        x_one, edge_index_one, batch_one = x_onein.x, x_onein.edge_index, x_onein.batch
        x_one = self.conv1(x_one, edge_index_one)
        return x_one

My input data come in batches of various sizes, and each graph has between tens of nodes to 1200 nodes. Because of this, I require the first input layer to be able to take in graphs of different sizes.
When I run the code, I get the error:
RuntimeError: Trying to create tensor with negative dimension -1: [256, -1]

This is strange since I followed the documentation:
in_channels (int or tuple) – Size of each input sample, or -1 to derive the size from the first input(s) to the forward method. A tuple corresponds to the sizes of source and target dimensionalities.
Any advice on this matter would be greatly appreciated.

Context:
I can’t reproduce the issue using:
x1 = torch.randn(4, 8)
x2 = torch.randn(2, 16)
edge_index = torch.tensor([[0, 1, 2, 3], [0, 0, 1, 1]])
conv = TransformerConv(in_channels=-1, out_channels=256)
out = conv(x1, edge_index)

so maybe your torch_geometric version is too old to support the lazy initialization of the layers (I don’t know when it was introduced)?


Reshape tensors your preserve variable information/structure

Question:
I have a multidimensional dataset, (1827, 5).
From that dataset I exctract the variable I want to predict and im left with my X and y variables as such:
X has size (1827, 4) and y has size (1827, 4).
Then i further split them into train and test datasets giving 20% to the test dataset. now the shapes I have are these:
> print(X_train.shape)
> print(y_train.shape)
> print(X_test.shape)
> print(y_test.shape)

torch.Size([1461, 4])
torch.Size([1461])
torch.Size([366, 4])
torch.Size([366])

The batch size I have to use (I cannot change that) is 50.
My question is, when reshaping a tensor so as to fit my model, shouldnt I take the original dimensions into account ? By dividing the train dataset into batches of 50, I will surely put data that belongs to the same observation into different batches,  simply because 50 is not divisible by 4 exactly.
My network architecture is this:
class NeuralNetwork(nn.Module):

    def __init__(self):
        super(NeuralNetwork, self).__init__()
        self.hidden1 = torch.nn.Linear(50, 25) # hidden layer
        self.hidden2 = torch.nn.Linear(25, 25) # hidden layer
        self.out = torch.nn.Linear(25, 1)      # output layer

    def forward(self, x):
        z = F.relu(self.hidden1(x)) # activation function for first hidden layer
        z = F.relu(self.hidden2(z)) # activation function for second hidden layer
        z = self.out(z)             # linear output
        return z

Im the first layer takes 50 as input because the batch size is 50, the output layer outputs 1 because im doing regression. The in-between I chose 25 because I read that a good number is the median between the input and output.
This problem came to be because I tried training the model with the data as is and I got this error:
mat1 and mat2 shapes cannot be multiplied (50x4 and 50x25)

I know that by changing the layers to output 4 it will be solved but Im wondering if its better to just reshape the data to (50, 25) instead.

Context:
elemeo:

Im the first layer takes 50 as input because the batch size is 50

This seems to be wrong, since the layer dimensions are not depending on the batch size.
You should defined the layers using their expected input features, not the number of samples they would see during training/inference.
Based on your initial description you are dealing with 1827 samples where each sample has 4 features. If that’s the case, use self.hidden1 = torch.nn.Linear(4, 25) and let the DataLoader create the batches each with a shape of [50, 4] (except the last one which might be smaller).


How to make current data loader more efficient

Question:
Hi, I have a working data loader, however while I am training it on the gpu cluster the average time taken to run per epoch is too much (approx. 25 mins). I am using PNG images for training with an average dimension of the image around (2500x5000). I want to make the training more faster. Please suggest what more can i do in order to do the training fast. Attached is the code of the data loader.
class MammographyCBISDDSM(Dataset):
def __init__(self, excel_file, category, transform=None):
    """
    Args:
        excel_file (string): Path to the excel file with annotations.
        category (string) : 'Classification' for Benign and Malignant. 'Subtypes' for Subtype Classification
        transform (callable, optional): Optional transform to be applied
    """
    self.mammography = pd.read_csv(excel_file,sep = ';')
    self.category = category
    self.transform = transform

def __len__(self):
    return len(self.mammography)

def class_name_to_labels(self, idx):
    if self.category == 'Classification':
        class_name = self.mammography.iloc[idx, 9]
        if class_name == 'MALIGNANT':
            labels = 0.0
        elif class_name == 'BENIGN':
            labels = 1.0
        elif class_name == 'BENIGN_WITHOUT_CALLBACK':
            labels = 2.0

    return labels

def __getitem__(self, idx):
    if torch.is_tensor(idx):
        idx = idx.tolist()

    img_folder = self.mammography.iloc[idx, 11]
    image_array=cv2.imread(img_folder)
    image_array = image_array.pixel_array
    image_array = image_array * 1.0 / image_array.max()
    image_array = torch.from_numpy(image_array.astype(np.float32))
    image_array = image_array.repeat(3, 1, 1)

    if self.transform:
        image_array = self.transform(image_array)

    labels = self.class_name_to_labels(idx)
    labels = torch.from_numpy(np.array(labels))
    return image_array, labels

Context: This great post 6 explains common issues causing a data loading bottleneck and has some suggestions to speed it up.


How to get one sample of dataset to train and to test

Question:
dataloader = datasets.CIFAR10
trainset = dataloader(root='./data', train=True, download=True,
                                     transform=transform_train)

Now I want to get one sample of trainset to train for debugging my code ,how to  write the code?

Context:
Based on your code, you should be able to get the first element by running this:
print(next(iter(trainset)))

You may find the examples on this page 1 helpful.


Dataloader creating data which is partially on CPU and GPU

Question:
I am running the code below to create data loaders for Graph data:
“”"
batch_size = 128
train_list =[]
for idx, batch in enumerate(zip(X_train[train_idx],
class_v[train_idx],
adj_train[train_idx])):
edge_index, _ = dense_to_sparse(torch.from_numpy(batch[2]).float())
train_list.append(Data(x = torch.from_numpy(batch[0]).float(),
y = torch.from_numpy(batch[1]).float(),
edge_index = edge_index
)
)
batch_train_loader = DataLoader(train_list, batch_size=batch_size)
val_list = []
for idx, batch in enumerate(zip(X_train_val[val_idx],
class_v_val[val_idx],
adj_train_val[val_idx])):
edge_index, _ = dense_to_sparse(torch.from_numpy(batch[2]).float())
val_list.append(Data(x = torch.from_numpy(batch[0]).float(),
y = torch.from_numpy(batch[1]).float(),
edge_index = edge_index)
)
batch_val_loader = DataLoader(val_list, batch_size=batch_size)
“”"
When I try to read this data. I get an error that data cannot be on two devices.
“”"
idx = 0
for data in batch_train_loader:
idx+=1
print(data.x.shape,data.y.shape,data.edge_index.shape)
if idx==3:
break
“”"
“”"
RuntimeError                              Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_17752/355153217.py in
1 idx = 0
----> 2 for data in batch_train_loader:
3     idx+=1
4     print(data.x.shape,data.y.shape,data.edge_index.shape)
5     if idx==3:
~\miniconda3\lib\site-packages\torch\utils\data\dataloader.py in next(self)
519             if self._sampler_iter is None:
520                 self._reset()
 → 521             data = self._next_data()
522             self._num_yielded += 1
523             if self._dataset_kind == _DatasetKind.Iterable and \
~\miniconda3\lib\site-packages\torch\utils\data\dataloader.py in _next_data(self)
559     def _next_data(self):
560         index = self._next_index()  # may raise StopIteration
 → 561         data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
562         if self._pin_memory:
563             data = _utils.pin_memory.pin_memory(data)
~\miniconda3\lib\site-packages\torch\utils\data_utils\fetch.py in fetch(self, possibly_batched_index)
45         else:
46             data = self.dataset[possibly_batched_index]
—> 47         return self.collate_fn(data)
~\miniconda3\lib\site-packages\torch_geometric\loader\dataloader.py in call(self, batch)
37
38     def call(self, batch):
—> 39         return self.collate(batch)
40
41
~\miniconda3\lib\site-packages\torch_geometric\loader\dataloader.py in collate(self, batch)
17         elem = batch[0]
18         if isinstance(elem, Data) or isinstance(elem, HeteroData):
—> 19             return Batch.from_data_list(batch, self.follow_batch,
20                                         self.exclude_keys)
21         elif isinstance(elem, torch.Tensor):
~\miniconda3\lib\site-packages\torch_geometric\data\batch.py in from_data_list(cls, data_list, follow_batch, exclude_keys)
61         Will exclude any keys given in :obj:exclude_keys."""
62
—> 63         batch, slice_dict, inc_dict = collate(
64             cls,
65             data_list=data_list,
~\miniconda3\lib\site-packages\torch_geometric\data\collate.py in collate(cls, data_list, increment, add_batch, follow_batch, exclude_keys)
74
75             # Collate attributes into a unified representation:
—> 76             value, slices, incs = _collate(attr, values, data_list, stores,
77                                            increment)
78
~\miniconda3\lib\site-packages\torch_geometric\data\collate.py in _collate(key, values, data_list, stores, increment)
142             incs = get_incs(key, values, data_list, stores)
143             if incs.dim() > 1 or int(incs[-1]) != 0:
 → 144                 values = [value + inc for value, inc in zip(values, incs)]
145         else:
146             incs = None
~\miniconda3\lib\site-packages\torch_geometric\data\collate.py in (.0)
142             incs = get_incs(key, values, data_list, stores)
143             if incs.dim() > 1 or int(incs[-1]) != 0:
 → 144                 values = [value + inc for value, inc in zip(values, incs)]
145         else:
146             incs = None
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
“”"
Not sure why this should happen. I am also seeing a considerable slowing in my training loop.

Context:
Are you setting the default tensor type to a CUDATensor or is any class pushing the data to the GPU (e.g. Data)?
Could you check the .device attribute of the tensors while iterating the train_list?


Filter out items from Dataset? “filter_pred” for any dataset

Question:
Hello,
For some datasets in torchtext, I see the parameter filter_pred  which filter out part of the dataset.
I’m using torchtext.Datasets which gives an IterableDataset. This iterator you can pass to the Dataloader. Is it possible to simply apply a filter like filter_pred?
This is my solution, which I don’t like, if you have something better, please share.
class MyIterableDataset(IterableDataset):
    def __init__(self, iterableDataset, pred_filter=None):
        self.iterableDataset = iterableDataset
        self.pred_filter = pred_filter

    def __iter__(self):
        def it(myDataIter):
            for x in myDataIter:
                if self.pred_filter and not self.pred_filter(x):
                    continue
                yield x
        myiter = iter(it(self.iterableDataset))
        return myiter

Context:
Does this work for you?
class MyIterableDataset(IterableDataset):

    def __init__(self, iterableDataset, pred_filter=None):
        self.iterableDataset = iterableDataset
        self.pred_filter = pred_filter

    def __iter__(self):
        return self

    def __next__(self):
        while True:
            item = next(self.iterableDataset)
            if self.pred_filter is None or self.pred_filter(item):
                return item


Question:
Hello there,
I want to make custom dataset or dataloader, just don’t know which is the best way to do it.
This is a self supervised task, where you want to blank out part of the input and then use it as a “label”.
Let’s say I have .npy files, each one of shape (600,30000), and I want to do the following:

Read the file,
pick a row from the 600,
take a slice around that row, say (32,30000),
return y=x[row] and x where x[row]=0 (blanked out).
The next batch should be another x,y pair where you just pick another row, and then repeat this for all the rows of the file, then move on to the net file.

After thoroughly reading the docs I thought that the best way to do that, is to make an Iterable Dataset, but whenever I try to increase the batch_size e.g. 56, in order to get a batch of (56,32,30000), this takes too long ~12’’ for a single batch which is an eternity of waiting for the GPU.
Code for the iterable dataset:
class MyIterableDataset(torch.utils.data.IterableDataset):
    def __init__(self, data_path, N_sub, batch_size, channel_min = 1700, channel_max = 2300):
        self.data_path = data_path
        self.filenames = [x for x in os.listdir(data_path) if x.endswith(".npy")]
        self.channel_min = channel_min
        self.channel_max = channel_max
        self.N_sub = N_sub
        self.batch_size = batch_size

    def sliding_window(self):
        for file in self.filenames:
            data = np.load(f"{data_path}{file}",mmap_mode='r')[self.channel_min:self.channel_max]
            for row in range(data.shape[0]):
                low_index = int(row - self.N_sub/2)
                high_index = int(row + self.N_sub/2)
                # if target is close to zero, then pick range [0, N_sub], target is not centered.
                if int(row - self.N_sub/2) <= 0:
                    low_index = 0
                    high_index = self.N_sub
                # if target is close to max channel, pick range [Nsub, max_channel], target is not centered again.
                if int(row + self.N_sub/2) >= self.channel_max-self.channel_min:
                    high_index = self.channel_max-self.channel_min
                    low_index = (self.channel_max - self.channel_min) - self.N_sub

                # Normalization, this causes minimal data leakage
                data = data/data.std()
                # Copy because assigning values to slices messes things up
                y_ = data[row].copy()

                # Zero out the target channel, the model will predict this.
                data[row] = 0
                x_ = data[low_index:high_index]
                # Keep only frequencies from f_min to f_max.
                # x_ = taper_filter(x_, self.f_min, self.f_max, self.sampleRate)
                # y_ = taper_filter(y_, self.f_min, self.f_max, self.sampleRate)
                x_ = torch.tensor(x_.astype(np.float32).copy())
                y_ = torch.tensor(y_.astype(np.float32).copy())
                yield x_, y_

    def __iter__(self):
        return self.sliding_window()

    def __len__(self):
        return len(self.filenames)

I thought of parallelizing this, I just can’t find an example related to this, so:

Any code/examples/advice you see fit is welcome,
Is this the way to go? Or am I overcomplicating things?

Thanks a lot in advance,
Have a great day!

Context: You can try multi-process data loading 4 and you may find the example 2 4 here helpful.


Batching a nested list of graphs in torch_geometric

Question: I am currently training a model which is a mix of graph neural networks and LSTM. However that means for each of my training sample, I need to pass in a list of graphs. The current batch class in torch_geometric supports batching with torch_geometric.data.Batch.from_data_list() but this only allows one graph for each data point. How else can I go about batching the graphs when each data consist of multiple graphs?

Context: Do you want to post the question/request to torch_geometric repo? We are not familiar with that.


Question:
Hi,
I was wondering if I could get a better understanding of data Augmentation in PyTorch. From what I know, data augmentation is used to increase the number of data points when we are running low on them. So we use transforms to transform our data points into different types. I am suing data transformation like this:
transform_img = transforms.Compose([
        transforms.RandomResizedCrop(224),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        # transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
])

class dataload(Dataset):

    def __init__(self, x, transform=None):
        self.data = x
        self.transform = transform

    def __len__(self):
        return len(self.data)

    def __getitem__(self, i):

        img = Image.open(self.data[i])
        # img = img.transpose((2, 0, 1))
        # img = torch.from_numpy(img).float()
        tmp = np.int32(filenames[i].split('/')[-1].split('_')[0][1])
        label = np.zeros(67)
        label[tmp] = 1
        label = torch.from_numpy(label).float()

        if self.transform:
            img = self.transform(img)

        return img,label

train_dataloader = dataload(filenames, transform=transform_img)

Now, it seems to work but I don’t get one thing. It does the transformation but it doesn’t increase the number of data points. I was hoping that each label would have 2 extra images since we are doing that transformation. But it doesn’t seem to do that. The total number of training samples is still the same.
So am I getting something wrong about augmentation or have I implemented this in the wrong way?

Context:
This way i’d call it alteration, not augmentation. Augmentation is when you are creating additional training samples. You need to move transformations to init, transform all x’es and add result to original data.
Also take a look at timm 1 library for the augmentations, cutmix and mixup implementations helped me a lot in recent project.


Loading specific optimizer tensors ont different devices using torch.load

Question:
Hi
I have a neural net model with optimizer state data saved on a pickled file (excuse if my terminology is imprecise) at a checkpoint.  The tensors for the model and the optimizer were all saved from the GPU, and when the checkpoint is loaded using torch.load, they are loaded on the GPU again.  However, when I load from the checkpoint, I would like some specific optimizer state tensors (e.g., the exponential moving average momentum tensor of the Adam optimizer, for example) to be loaded into CPU instead of GPU.  I have found very little documentation on the map_location option in torch.load to explain how to write a function that does this.  Can you  please provide an example for how to do this for one or more tensors?  You can assume that the optimizer tensors to be loaded into CPU are state[‘exp_avg’] and state[‘exp_avg_sq’] for definiteness.
Thanks!

Context:
You can pass map_location='cpu' to torch.load to load the object to the CPU.
Here is a small example showing the usage:
# save optimizer.state_dict() with CUDATensors
model = models.resnet18().cuda()
optimizer = torch.optim.Adam(model.parameters(), lr=1.)
out = model(torch.randn(1, 3, 224, 224).cuda())
out.mean().backward()
optimizer.step()

torch.save(optimizer.state_dict(), 'opt.pth')
optimizer.state_dict()

# in a new script make sure the GPU is not visible and load the state_dict to the CPU
import os
os.environ['CUDA_VISIBLE_DEVICES'] = ""
import torch
import torchvision.models as models

torch.cuda.get_device_name()
# > RuntimeError: No CUDA GPUs are available

model = models.resnet18()
optimizer = torch.optim.Adam(model.parameters(), lr=1.)
optimizer.load_state_dict(torch.load('opt.pth'))
# > RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.
optimizer.load_state_dict(torch.load('opt.pth', map_location='cpu')) # works


Question:
New to python. How can I modify the class to filter files in the folder with a string. Right now it returns all files in folder_containing_the_content_folder which could be millions of items. I would like to isolate files that contain a specific string, for example, isolate all files that contain ‘v_1234_frame’:
# Image loader
transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Lambda(lambda x: x.mul(255))
])
image_dataset = utils.ImageFolderWithPaths(folder_containing_the_content_folder, transform=transform)
image_loader = torch.utils.data.DataLoader(image_dataset, batch_size=batch_size)

The class that works requires a modification to filter file names that contain ‘v_1234_frame’:
class ImageFolderWithPaths(datasets.ImageFolder):
"""Custom dataset that includes image file paths.
Extends torchvision.datasets.ImageFolder()
Reference: https://discuss.pytorch.org/t/dataloader-filenames-in-each-batch/4212/2
"""

# override the __getitem__ method. this is the method dataloader calls
def __getitem__(self, index):
    # this is what ImageFolder normally returns
    original_tuple = super(ImageFolderWithPaths, self).__getitem__(index)

    # the image file path
    path = self.imgs[index][0]

    # make a new tuple that includes original and the path
    tuple_with_path = (*original_tuple, path)
    return tuple_with_path

I am learning python and just can’t seem to come up with the solution. Hope you can help/suggest a change to the class or calling method.

Context: You could e.g. create a custom find_classes method and pass it to DatasetFolder as described here 1.


How to use multi-thread in pytorch dataloader with pybind release-gil functions?

Question:
Half of my project written in C++ with pybind nogil wrapper and unpickleable c++ class, so I can only use multi-thread loader instead of multi-process.
Does pytorch implement multi-thread for loader? If not, why?

Context:
General reason is multithreading is not fast in python. For the use case of DataLoader, most of users create Dataset using pure python code. Then, there won’t be any performance gain if we provide that.
But, we are working on new DataLoader-DataPipe, and hopefully we can provide multithreading DataLoader for you in the short future. And, you can use that to call a c++ function without python gil.


Focal loss performs worse than cross-entropy-loss in clasification

Question:
Hello,
I am working on a CNN based classification.
I am using torchvision.ImageFolder to set up my dataset then pass to the DataLoader and feed it to
pretrained resnet34 model from torchvision.
I have a highly imbalanced dataset which hinders model performance.
Say ‘0’: 1000 images, ‘1’:300 images.
I know I have two broad strategies: work on resampling (data level) or on loss function(algorithm level).
I first tried to change the cross entropy loss to custom FocalLoss. But somehow I am getting even worse performance like below:

my training function looks like this can anybody tell me what I am missing out or doing wrong?
def train_model(model, data_loaders, dataset_sizes, device, n_epochs=20):

    optimizer = optim.Adam(model.parameters(), lr=0.0001)
    scheduler = lr_scheduler.StepLR(optimizer, step_size=30, gamma=0.1)
    loss_fn = FocalLoss().to(device)
    history = defaultdict(list)
    best_accuracy = 0
    for epoch in range(n_epochs):
        print(f'Epoch {epoch + 1}/{n_epochs}')
        print('-' * 10)
        train_acc, train_loss = train_epoch(
          model,
          data_loaders['train'],
          loss_fn,
          optimizer,
          device,
          scheduler,
          dataset_sizes['train']
        )
        print(f'Train loss {train_loss} accuracy {train_acc}')
        val_acc, val_loss = eval_model(
          model,
          data_loaders['val'],
          loss_fn,
          device,
          dataset_sizes['val']
        )
        print(f'Val   loss {val_loss} accuracy {val_acc}')
        print()
        history['train_acc'].append(train_acc)
        history['train_loss'].append(train_loss)
        history['val_acc'].append(val_acc)
        history['val_loss'].append(val_loss)
        if val_acc > best_accuracy:
            torch.save(model.state_dict(), 'best_model_state.bin')
            best_accuracy = val_acc
    print(f'Best val accuracy: {best_accuracy}')
    model.load_state_dict(torch.load('best_model_state.bin'))
    return model, history

The custom FocalLoss function from the web looks like below (sorry, I forgot the reference):
class FocalLoss(nn.Module):
    #WC: alpha is weighting factor. gamma is focusing parameter
    def __init__(self, gamma=0, alpha=None, size_average=True):
    #def __init__(self, gamma=2, alpha=0.25, size_average=False):
        super(FocalLoss, self).__init__()
        self.gamma = gamma
        self.alpha = alpha
        if isinstance(alpha, (float, int)): self.alpha = torch.Tensor([alpha, 1 - alpha])
        if isinstance(alpha, list): self.alpha = torch.Tensor(alpha)
        self.size_average = size_average

    def forward(self, input, target):
        if input.dim()>2:
            input = input.view(input.size(0), input.size(1), -1)  # N,C,H,W => N,C,H*W
            input = input.transpose(1, 2)                         # N,C,H*W => N,H*W,C
            input = input.contiguous().view(-1, input.size(2))    # N,H*W,C => N*H*W,C
        target = target.view(-1, 1)

        logpt = F.log_softmax(input, dim=1)
        logpt = logpt.gather(1,target)
        logpt = logpt.view(-1)
        pt = logpt.exp()

        if self.alpha is not None:
            if self.alpha.type() != input.data.type():
                self.alpha = self.alpha.type_as(input.data)
            at = self.alpha.gather(0, target.data.view(-1))
            logpt = logpt * at

        loss = -1 * (1 - pt)**self.gamma * logpt
        if self.size_average: return loss.mean()
        else: return loss.sum()

Context:
Take a look at the original paper 1 for more insight.
For focal loss:

" In practice α may be set by inverse class frequency or treated as a hyperparameter to set by cross validation"

Also, your problem is not highly imbalance. You can use weighted cross entropy 1 and get a good performance.


Applying different data augmentation per image in a mini-batch

Question:
As far as I know, the random transformations (e.g. random crop, random resized crop, etc.) from torchvision.transforms module apply the same transformations to all the images of a given batch.
Is there any efficient way to apply different random transformations for each image in a given mini-batch?
Thanks in advance.

Context:
I think you would need to apply the random transformations on each sample and could use e.g. transforms.Lambda for it:
# setup
x = torch.zeros(3, 10, 10)
x[:, 3:7, 3:7] = 1.

# same transformation on each sample
transform = transforms.RandomCrop(size=5)
y1 = transform(x)
print(y1)

# apply transformation per sample
y2 = transforms.Lambda(lambda x: torch.stack([transform(x_) for x_ in x]))(x)
print(y2)


How to define a pytorch class Dataset which take dataset with and without label

Question:
Hello, I ma trying to define a Dataset class which is suppoose to treat dataset with label and without label. But I ma getting and error
Item = {key: torch.tensor(val[idx]) for key, val in self.encodings.items()} AttributeError: ‘list’ object has no attribute ‘items’

code
# Create torch dataset
class Dataset(torch.utils.data.Dataset):
	def __init__(self, encodings, labels=None):
		self.encodings = encodings
		self.labels = labels

	def __getitem__(self, idx):
		item = {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}
		if self.labels:
			item["labels"] = torch.tensor(self.labels[idx])
			#print(item)
		return item

	def __len__(self):
		print(len(self.encodings["input_ids"]))
		return len(self.encodings["input_ids"])

# prepare dat for classification

tokenizer = FlaubertTokenizer.from_pretrained(model_name)
	print("Transform xml file to pandas series core...")
	text, file_name = transform_xml_to_pd(file)  # transform xml file to pd

	# Xtest_emb, s = get_flaubert_layer(Xtest['sent'], path_to_model_lge)  # index 2 correspond to sentences
	#print(text)

	print("Preprocess text with spacy model...")
	clean_text = make_new_traindata(text['sent'])
	#print(clean_text[1])  # clean text ; 0 = raw text ; and etc...

	X = list(clean_text)
	X_text_tokenized = []

	for x in X:
		#print(type(x))
		x_encoded = tokenizer(str(x), padding="max_length", truncation=True, max_length=512)
		#print(type(x_encoded))
		#print(x_encoded)
		X_text_tokenized.append(x_encoded)

	#print(type(X_text_tokenized))

	X_data = Dataset(X_text_tokenized)

	print(type(X_data))
	print(X_data['input_ids'])
Error :slight_smile:

  File "/scriptTraitements/classifying.py", line 153, in __getitem__
    item = {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}
AttributeError: 'list' object has no attribute 'items'

Any idea  ?
furthermore How can I access element of the corpus pass to the Dataset class. When printing it , I get only this : <main.Dataset object at 0x7fc93bec1df0>

Context: it says that you’re not suppose to use .items ( a dict method) on list object.


Transformations. How do I keep it synchronized with target if target is a pixel coordinate in the image?

Question:
I create a neural network, which should search the image for objects (faces away from the camera). In fact I need 4 coordinates, for the example I attach the image in which these coordinates are marked. There is a good set of transformers in pytorch, but the problem is that if the transformation is related to image distortion, you need to change the target respectively, otherwise the coordinates will not coincide with the desired points. What is the best way to do this? Maybe PyTorch already has tools implemented for such purposes?
This is what the target looks like for the image below - it is a set of coordinates
[[1253  368]
 [1251  589]
 [1386  579]
 [1368  800]]

Translated with www.DeepL.com/Translator (free version)

Снимок экрана 2021-12-09 080614720×408 321 KB

Context:
you can use :

      albumentations.ai

Albumentations Documentation - Using Albumentations to augment keypoints

  Albumentations: fast and flexible image augmentations


How to create Dataset out of Matlab Array?

Question:
Hi guys,
I am absolutely new here and therefore have little or no experience with PyTorch.
I would like someone to be able to tell me the best way to use a data array from Matlab as a data set, preferably with a normalization of the data, e.g. in the range [0, 1].
Here a little more detail about my data:
I have a two-dimensional array, with the rows corresponding to frames from several spectrograms. The size of the array is 60000 x 1025, i.e. 60000 frames with 1025 frequency bins. Since I use an auto encoder, my input data is also my target data.
I already know that I can use mat73.loadmat (path) to read my Matlab data as an array in Python. Now the only question is how can I create a normalized dataset from this?
I hope you guys can help me.
Thanks in advance

Context:
if you are loading your data to a numpy array, do as follow:
dataset = torch.from_numpy(data)

and if your data is loading to python list:
dataset = torch.tensor(data)

for data normalization:
dMin  = (dataset.min(0,keepdim=True))[0]
dMax = (dataset.max(0,keepdim=True))[0]
NDataset = (dataset - dMin)/(dMax - dMin)

then you need to define data loader:
dl = torch.utils.data.DataLoader(NDataset,batch_size=100,shuffle=True)

for i in range(epoch):
    for x in dl:
         #do something with it.


Torch.save()/can not save the whole tensor if the tensor takes more than 10 GB storage

Question:
What should I do to deal with this problem?
Thank you very much in advance!

Context: I’m not sure if this issue might be Windows-specific, but could you update PyTorch to the latest nightly release and check, if this might have been an already known and fixed issue, please?


Question: I ran into WARNING:root:Torch AMP is not available on this platform consistently when running SlowFast on 8 V100s. And I’m not even enabling amp. Any idea where this warning comes from?

Context: I haven’t seen this warning before and searching for it yields a result in fairseq 4. The warning “namespace” also doesn’t seem to be created in PyTorch. Are you using fairseq or another library depending on PyTorch, which might raise this warning?


Deterministic training when using mixed-precision

Question:
TL;DR: After using torch.cuda.amp, I have deterministic training, even though I set torch.backends.cudnn options as deterministic=False, benchmark=False, etc.
Environment

2080Ti (CUDA 11.2, Driver 460.91.03)
PyTorch 1.11.0.dev20211127
Python 3.9.7

I experimented with this minimal MNIST example 1 and reproduced the nondeterminism across training runs (i.e. different epoch losses when I train from scratch multiple times). The source of nondeterminism is GPU operations since the random seeds are fixed.
If I set torch.backends.cudnn.deterministic=True, I see deterministic training like the original author.
However, if I use amp for mixed-precision training, I also see determinism even without deterministic=True.
Has anyone seen similar cases or have insights into how using amp could remove nondeterminism?
Here’s a writeup with further details on experiments and code. 1
Thanks!

Context:
juliustao:

Has anyone seen similar cases or have insights into how using amp could remove nondeterminism?

Non-deterministic results could be caused by the actual kernel selection and amp could be using e.g. deterministic cuDNN kernels.


Question:
Hello，
I tried to reduce communication bottleneck by using auto mixed precision training. I also used torch.profiler to check the training performance.
In the timeline, I noticed that the all-reduce kernel is ncclAllReduceRingLLKernel_sum_f32.

Does it mean the fp16 gradients are casted to fp32 to be all-reduced?  Does something like ncclAllReduceRingLLKernel_sum_f16 exist?
My GPU is P100. PyTorch version is 1.9.1+cu111.

Context: The parameters are stored in float32 using the automatic mixed-precision util. and thus also the gradients. NCCL thus communicates them in float32, too. If you are calling .half() on the model directly and thus apply a pure float16 training, NCCL should communicate in float16.


Optimizer.step() – ok; scaler.step(optimizer): No inf checks were recorded for this optimizer

Question:
I am getting AssertionError: No inf checks were recorded for this optimizer. in "/torch/cuda/amp/grad_scaler.py", line 291 when mixed-precision is used in this weird example below. However, if no mixed-precision is used pytorch doesn’t complain (toggle USE_HALF_PRECISION = True).
I am using PyTorch 1.6.0 (python 3.7, cuda 10.2.89, cudnn 7.6.5. – everything is in conda binaries). Here is the MWE.
import torch
from torch import nn
from torch.cuda.amp.autocast_mode import autocast
from torch.cuda.amp.grad_scaler import GradScaler

class Identity_with_weights(nn.Module):
    '''For example a KNN algorithm which returns a closest entry from a database for x. Weights are needed
    for a seamless inclusion of knn baseline to a set of baseline which do have some parameters. Otherwise
    you would need to change the code (remove optimizer, backward pass etc) just for knn which is not neat.'''
    def __init__(self):
        super(Identity_with_weights, self).__init__()
        self.__hidden__ = torch.nn.Linear(1, 1, bias=False)

    def forward(self, x):
        # we need it to be able to call backward on the loss which uses x (outputs).
        # Nothing will happen in these examples as it propagates to the input which is not used anywhere else
        x.requires_grad = True
        return x

if __name__ == "__main__":
    # config
    USE_HALF_PRECISION = True
    device = torch.device('cuda:0')

    # define model
    model = Identity_with_weights()

    # define training things
    criterion = nn.L1Loss()
    optimizer = torch.optim.Adam(model.parameters())

    # for amp
    scaler = GradScaler()

    # targets are exactly the same as inputs, i.e. for reconstruction
    inputs = torch.rand(8, 1)
    targets = inputs.clone().detach()

    # send to device
    model = model.to(device)
    inputs = inputs.to(device)
    targets = targets.to(device)

    # we don't need it for the sake of this example, but let's have it here anyway.
    optimizer.zero_grad()

    # since outputs are going to be f16 and targets are f32, criterion will output non zero loss
    if USE_HALF_PRECISION:
        targets = targets.half()
        inputs = inputs.half()

    # autocasting ops inside of the context manager
    with autocast(USE_HALF_PRECISION):
        outputs = model(inputs)
        loss = criterion(outputs, targets)

    print(loss)

    # scaling loss if using half precision
    if USE_HALF_PRECISION:
        scaler.scale(loss).backward()
        scaler.step(optimizer) ## ERROR HERE
        scaler.update()
    else:
        loss.backward()
        optimizer.step()

I think I am doing something wrong here. What does it complain about?

Context:
Scaler (optimizer) looks for parameters used in the graph which is empty, hence, the error.
This specific example was solved by replacing
x.requires_grad = True
with
x = x + 0 * self.__hidden__(x)


Override AMP casting during bfloat16 training

Question:
I’m trying to train a PyTorch model on an A100 with bfloat16 mixed precision.
torch AMP maintains a list of operations deemed numerically unstable in float16 (Automatic Mixed Precision package - torch.cuda.amp — PyTorch 1.10.1 documentation) and automatically casts the inputs to those operations to float32, even if you’re running AMP with bfloat16. Is there a way to globally tweak that list to prevent inputs to certain operations from being cast to full precision without manually enabling/disabling AMP every time they’re called? I’d like e.g. all softmax calls throughout my model to run using bfloat16.

Context: I’m unsure if there is an easy way to “tune” the autocast list (@mcarilli might know), but for now you could disable autocast for the desired operations, transform your data to bfloat16() and call the operations directly.


Switching between mixed-precision training and full-precision training after training is started

Question: Hi, using apex.amp or torch amp, is it possible that I switch between mixed precision training and full precision training after the training is started? For example, I might want the first 100 iterations to be trained with full precision, and switch to the mixed precision mode after the 100 iterations. Is this possible?

Context: Yes, you could use the enabled arguments in torch.cuda.amp.autocast as well as in the creation of the GradScaler and switch them to True/False in your training run.


Increased GPU memory usage on GPU 0 when using AMP

Question:
When using AMP, there is a large GPU memory increase on GPU 0. Below is the pseudo code to show my training pipeline. Before running self.scaler.update(), 19G memory is used on each of the 8 GPUs. After the line is done, GPU 0 memory usage will be 29G. This happens in the first training iteration. Is it normal? Or is there any method to distribute the extra memory usage evenly to all the GPUs?
More info:

I am using DistributedDataParallel.
Pretrained checkpoint is loaded to different GPUs.

class Trainer:
    def __init__():
        self.optimizer = ...
        self.scaler = GradScaler()

    def train(input):
        output = model(input)
        loss = get_loss(output)
        self.scaler.scale(loss).backward()
        self.scaler.step(self.optimizer)
        self.scaler.update()
        self.optimizer.zero_grad()

Context: How did you measure the memory? Is the increase purely in the cache or in the allocated memory? Were you seeing OOM issues when using AMP?


The performance gap between torch.cuda.amp and nviddia-apex

Question:
In the past I use nviddia-apex O1 mode to train my model. And the torch.cuda.amp seems to save more memory. I compared apex and torch.cuda.amp respectively for 1000 iterations, and find the performance gap on val set was 8 (mIoU for Semantic segmentation). The model with apex was 30 mIoU and with torch.cuda.amp was 22 mIoU. Is this normal ?
Here are some of the key implementations::
def __init__(self,):
    self.scaler = GradScaler()
def step(X, label)
    with autocast(enabled=True):
        _, output = self.forward(input=X)
        outputUp = F.interpolate(output, size=X.size()[2:], mode='bilinear', align_corners=True)
        loss = self.loss_source(input=outputUp, target=label)
    self.amp_backward(loss, self.BaseOpti)

And the amp_backward function was:
def amp_backward(self, loss, optim):
    self.scaler.scale(loss).backward()
    self.scaler.step(optim)
    self.scaler.update()

Context: No, an accuracy loss is unexpected. Which PyTorch version are you using and which model?


Question:
Hi, all.
I want to try apply AMP for the two optimizers .
I just added all losses to total_loss.
Below codes are correctly optimize the two optimizers?
optimizer.zero_grad()
optimizer2.zero_grad()
scaler.scale(total_loss).backward()
scaler.step(optimizer)
scaler.step(optimizer2)
scaler.update()

Thanks in advance.

Context: Yes, your code should work. The docs 1 also give an example which fits your use case.


Mixed precision training is so slow when deterministic=True

Question:
Hi folks, is there a way to set deterministic=True without get slow training using mixed precision ? is it a bug?
Here are my benchmarks using RTX 3070 (pytorch 1.10 - cuda 11.3):
deterministic=False     43.032ms per iter  vs
deterministic=True      354.005ms per iter
import torch
import torch.nn as nn
from torch.cuda.amp import autocast, GradScaler
import time
from torchvision.models import mobilenet_v2

class model(nn.Module):
    def __init__(self):
        super(model, self).__init__()
        self.mobilenet = mobilenet_v2()
        self.linear = nn.Sequential(nn.Linear(1000, 250),
                                    nn.Linear(250, 64),
                                    nn.Linear(64, 32),
                                    nn.Linear(32, 10),
                                    nn.ReLU())

    def forward(self, x):
        out = self.mobilenet(x)
        out = self.linear(out)
        return out

x = torch.randn(32, 3, 224, 224).cuda(0)
y = torch.randint(0, 10, (32,)).cuda(0)
m = model().cuda(0)
criterion = nn.CrossEntropyLoss()
optim = torch.optim.Adam(m.parameters(), 1e-4)
scaler = GradScaler()

nb_iters = 100

def train16(nb_iters=100, deterministic=False):
    torch.backends.cudnn.deterministic = deterministic
    torch.cuda.synchronize()
    t0 = time.time()
    for _ in range(nb_iters):
        optim.zero_grad()
        with autocast(enabled=True):
            output = m(x)
            loss = criterion(output, y)
        scaler.scale(loss).backward()
        scaler.step(optim)
        scaler.update()
    loss.detach()
    torch.cuda.synchronize()
    t1 = time.time()
    print('{:.3f}ms per iter'.format((t1 - t0)/nb_iters * 1000.))

train16(nb_iters, deterministic=True)
train16(nb_iters)

Thanks in advance!

Context: No, as deterministic algorithms could yield a slower performance to guarantee deterministic results due to the internal kernel implementation.


RuntimeError: expected scalar type Float but found Half in deform_conv2d

Question:
Hi, I have a huge network with some Deformable CNNs (torch.ops.torchvision.deform_conv2d) in it. I am using apex 0.1, Cuda 11, and torch 1.11. I have set the optimization level to O1 for mixed-precision training.
I have used nn.DataParallel and also cast the model to GPU properly. Unfortunately, I have this error. Could you please help me fix this?
File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/container.py", line 141, in forward
    input = module(input)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "<path>/model/dcn.py", line 56, in forward
    x = torchvision.ops.deform_conv2d(input=x,
  File "/opt/conda/lib/python3.8/site-packages/torchvision/ops/deform_conv.py", line 89, in deform_conv2d
    return torch.ops.torchvision.deform_conv2d(
RuntimeError: expected scalar type Float but found Half

Thanks!

Context:
I guess you might be using an older torchvision release, since the operation is available via torchvision.ops.DeformConv2d and this PR 1 added autocast support ~1 year ago.
This code snippet works fine for me:
from torchvision import ops

deform_layer = ops.DeformConv2d(in_channels=3, out_channels=64, kernel_size=3).cuda()
x = torch.randn((1,3,9,9)).cuda()
offset = torch.randn(1,3*3*2,7,7).cuda()
with torch.cuda.amp.autocast():
    out = deform_layer(x, offset)


Question:
Hi. I’m benchmarking automatic mixed precision vs. default mode (float32). I’m getting a speed-up but, the memory usage is the same, if not higher. I’m running Pytorch’s tutorial with minimal change:
import torch, time, gc
import os

os.environ["CUDA_VISIBLE_DEVICES"] = "0"

# Switching between two
use_amp = True
# use_amp = False

start_time = None

def start_timer():
    global start_time
    gc.collect()
    torch.cuda.empty_cache()
    torch.cuda.reset_max_memory_allocated()
    torch.cuda.synchronize()
    start_time = time.time()

def end_timer_and_print(local_msg):
    torch.cuda.synchronize()
    end_time = time.time()
    print("\n" + local_msg)
    print("Total execution time = {:.3f} sec".format(end_time - start_time))
    print(f'Memory allocated {torch.cuda.memory_allocated() // (1024**2)} MB')
    print(f'Max memory allocated {torch.cuda.max_memory_allocated() // (1024**2)} MB')
    print(f'Memory reserved {torch.cuda.memory_reserved() // (1024**2)} MB')
    print(f'Max memory reserved {torch.cuda.max_memory_reserved() // (1024**2)} MB')

def make_model(in_size, out_size, num_layers):
    layers = []
    for _ in range(num_layers - 1):
        layers.append(torch.nn.Linear(in_size, in_size))
        layers.append(torch.nn.ReLU())
    layers.append(torch.nn.Linear(in_size, out_size))
    return torch.nn.Sequential(*tuple(layers)).cuda()

epochs = 1
num_batches = 50
batch_size = 512 # Try, for example, 128, 256, 513.

in_size = 4096
out_size = 4096
num_layers = 16

# in_size = 8192
# out_size = 8192
# num_layers = 32

data = [torch.randn(batch_size, in_size, device="cuda") for _ in range(num_batches)]
targets = [torch.randn(batch_size, out_size, device="cuda") for _ in range(num_batches)]

loss_fn = torch.nn.MSELoss().cuda()

net = make_model(in_size, out_size, num_layers)
opt = torch.optim.SGD(net.parameters(), lr=0.001)

scaler = torch.cuda.amp.GradScaler(enabled=use_amp)

start_timer()
for epoch in range(epochs):
    for input, target in zip(data, targets):
        with torch.cuda.amp.autocast(enabled=use_amp):
            output = net(input)
            loss = loss_fn(output, target)
        scaler.scale(loss).backward()
        scaler.step(opt)
        scaler.update()
        opt.zero_grad() # set_to_none=True here can modestly improve performance
message = "Mixed precision:" if use_amp else "Default precision:"
end_timer_and_print(message)

Outputs:
Default precision:
Total execution time = 3.553 sec
Memory allocated 2856 MB
Max memory allocated 3176 MB
Memory reserved 3454 MB
Max memory reserved 3454 MB
# nvidia-smi shows 4900 MB

Mixed precision:
Total execution time = 1.652 sec
Memory allocated 2852 MB
Max memory allocated 3520 MB
Memory reserved 3646 MB
Max memory reserved 3646 MB
# nvidia-smi shows 5092

When I try to saturate the GPU (RTX 6000 with 24 GB memory) using different hyperparameters, default mode works, but AMP goes out of memory:
in_size = 4096
out_size = 4096
num_layers = 16

Outputs:
Default precision:
Total execution time = 29.503 sec
Memory allocated 18002 MB
Max memory allocated 19282 MB
Memory reserved 19284 MB
Max memory reserved 19284 MB
# nvidia-smi shows 20730 MB

# Mixed precision goes out of memory:
RuntimeError: CUDA out of memory. Tried to allocate 256.00 MiB (GPU 0; 23.65 GiB total capacity; 22.08 GiB already allocated; 161.44 MiB free; 22.08 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Torch version: 1.10.0.dev20210630+cu113
CUDA version: 11.3
GPU model: NVIDIA Quadro RTX 6000
Thanks in advance!

Context:
Hi, I’ve just try amp with pytorch yesterday with a Pascal gtx 1070.
I just which to “extend the gpu vram” using mixed precision.
Following the tutorial and increasing different parameters i saw that mixed precision is slower (for the Pascal GPU which seems normal) but the memory usage is higher with that GPU.
To verify that the : torch.cuda.max_memory_allocated() value represents really the memory usage, i found parameters for which the fp32 finishs the job but the mixed precision gives a RuntimeError: CUDA out of memory error.
I just try different values for the batch size and find a special value of 3860 that gives memory error only for the mixed precision.
In that case : the script shows : Default precision:
Total execution time = 42.469 sec
Max memory used by tensors = 7092684288 bytes
File “C:\Users\33661\miniconda3\envs\torch\lib\site-packages\torch\autograd_init_.py”, line 156, in backward
allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag
RuntimeError: CUDA out of memory. Tried to allocate 64.00 MiB (GPU 0; 8.00 GiB total capacity; 6.59 GiB already allocated; 0 bytes free; 6.73 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
It was pytorch 1.10 with cuda 11.3 and cudnn 8.2
I try with pytorch 1.7.1 ; cuda 10.1 and cudnn 7.6 with analog results.
I would like to know which version of pytorch, cuda and cudnn (maybe not important?) allow to show a reduction of memory usage with the tutorial script which is well done.
Maybe this problem is due to cuda version?
I try some basic convolutional neural network and always the mixed precision use more gpu vram.
I wonder if the mixed precision for the moment does its principle job : allow to use bigger networks and batchsize.
I will try older pytorch versions and will  try to use apex.amd to see if theses problem always remained.
I test tensorflow.keras for small convolutional nets with mnist data and always find memory usage reduction with mixed precision. The speed could  decrease  for small networks with the gtx 1070 without tensor core.
I find the pytorch tutorial marvelous and love pytorch in general but if i need mixed precision i will use surely another framework.


Where I can see whether and operation autocasts or not?

Question:
Hi,
simple question: Can someone point me to the part of the documentation where I can see how/whether an operation autocasts to fp16 when run with AMP or whether it keeps using fp32. Or can this only be seen from the source code?
Best,
Tim

Context:
Hi,
See the doc on autocast about autocast operation reference, in particular the part about Op-Specific Behavior where there is such a list. Is it what you are looking for?
Best regards,
Thomas


Torch.cuda.amp blocks gradient computation

Question:
Hi !
I had an issue while using torch.cuda.amp.autocast(). In the context manager I called torch.no_grad() and it blocked gradient computation in the rest of the context manager.
Example on how to reproduce :
import torch
import torch.nn as nn
net = nn.Linear(100, 10).cuda()
input = torch.randn(32, 100, device='cuda')
target = torch.randn(32,10, device='cuda')
crit = nn.MSELoss()

with torch.cuda.amp.autocast():
    with torch.no_grad():
        out = net(input)

    new_out = net(input)
    loss = crit(new_out, target)

Then loss will not have a grad_fn
But if we would not use autocast :
import torch
import torch.nn as nn
net = nn.Linear(100, 10).cuda()
input = torch.randn(32, 100, device='cuda')
target = torch.randn(32,10, device='cuda')
crit = nn.MSELoss()

with torch.cuda.amp.autocast(False):
    with torch.no_grad():
        out = net(input)

    new_out = net(input)
    loss = crit(new_out, target)

Then loss has a grad_fn.
Is it an expected behavior ?
Thanks for any help !

Context:
You would have to exit the autocast context before running the second forward pass as described here 3 due to the internal caching.
CC @vadimkantorov as we’ve discussed more general use cases and this one might be important for you.


Can autocast context manager be used around all of training loop?

Question: Often one has a method like train_one_epoch. Can I just use autocast around the call of this method? Or must we wrap only the call to the model after data iteration / move to device is done?

Context:
I would stick to the recommended usage of moving the model’s forward and loss calculation into the autocast context. These pieces from the docs might be interesting to check your use case:

autocast should wrap only the forward pass(es) of your network, including the loss computation(s). Backward passes under autocast are not recommended. Backward ops run in the same type that autocast used for corresponding forward ops.

Floating-point Tensors produced in an autocast-enabled region may be float16. After returning to an autocast-disabled region, using them with floating-point Tensors of different dtypes may cause type mismatch errors. If so, cast the Tensor(s) produced in the autocast region back to float32 (or other dtype if desired). If a Tensor from the autocast region is already float32, the cast is a no-op, and incurs no additional overhead.

Besides that autocast uses an internal cache to avoid re-casting constant tensors, which would never be freed if you are wrapping the entire code into an autocast region and you might thus have to either disable it or could suffer from increasing memory usage.


Why is closure not supported in GradScaler ?

Question:
In the step function of GradScaler, if a closure is given as a member of kwargs, RuntimeError occurs by the following codes.

      github.com

    pytorch/pytorch/blob/04fe4382ec8e6031d353530d6b8446ac9c9f507e/torch/cuda/amp/grad_scaler.py#L312-L313 2

          if "closure" in kwargs:
              raise RuntimeError("Closure use is not currently supported if GradScaler is enabled.")

The error message says “not currently supported”. Are there any plans to support this feature? Or Would you tell me reasons why closure isn’t supported? If possible, I want to try to make a patch to solve this issue.
My motivation is to support SAM with native amp easily.

Context:
Making closures work with dynamic gradient scaling (specifically, the fact that dynamic gradient scaling occasionally skips optimizer.step() if any grads were inf/nan) is tricky, and we haven’t heard any use cases that absolutely needed it (LBFGS is the only one I’m aware of and no one’s asked for that).
Can you implement SAM without a closure like this?

      GitHub

GitHub - davda54/sam: SAM: Sharpness-Aware Minimization (PyTorch) 3

  SAM: Sharpness-Aware Minimization (PyTorch). Contribute to davda54/sam development by creating an account on GitHub.


Does amp autocast cache fp16 copies of model parameters?

Question:
I’m thinking of using autocast for speeding up processing of frozen backbone. If I use autocast context manager in forward, will it every time spend time on conversion of model weights?
How do I ensure it does conversion of model parameters fp32->fp16 exactly once?
My alternative is to manually convert frozen backbone’s parameters to fp16, and maybe not use any autocast at all, but I assume autocast is a better bet because it has special cases for ops that still must be run in fp32.

Context:
Yes, autocast uses caching internally unless you manually disable it and the checks can be found here 2.

 vadimkantorov:

How do I ensure it does conversion of model parameters fp32->fp16 exactly once?

It’ll be done automatically for you if the parameter is found in the cache (which won’t be the case if e.g. you’ve modified it).


Question:
Geting this with pytorch 1.8. It also says

you may get better performance by installing NVIDIA’s apex

But this confuses me because I thought apex was integrated into pytorch nowadays?
Could I fix this problem just by upgrading pytorch?

Context:
As far as I understand, the AMP (now from PyTorch) and the fused kernels (still from apex) are largely independent even if the module is called amp_C, so you would only need to install apex.
In general my impression was that fairseq is relatively well maintained and the people have good access to PyTorch know-how, so their code would do the right thing.
Best regards
Thomas


Do I need to save the state_dict oof GradScaler?

Question: Do I need to save the state_dict of torch.cuda.amp.GradScaler and reload it to resume training? The docs say it dynamically estimates the scale factor each iteration, so I never saved it. So, will model.load_state_dict and optimizer.load_state_dict suffice?

Context:
If you want to restore the last scale factor (as well as the backoff and growth factor, if changed), then you should restore its state_dict.
Your training should also work without restoring the gradient scaler, but will most likely not reproduce the same results as a run without interruptions, as the new gradient scaler could skip iterations at different steps.


Training GANs using automatic mixed precision？

Question:
Hi, guys.
I encountered an overflow problem when deploying the GAN model(trained in fp32) to fp16 precision. And I decided to give it a try after seeing amp 1. I’m wondering how G and D are trained in amp context. (Only G needs to be deployed in fp16 while D is only used when training)

Context:
The linked doc shows an example how to use amp for multiple models, losses, and optimizers 6.
Did you follow these steps or did you use a custom approach?


Question:
Hello everyone,
I train custom Bert model and observe the following problems.
These points are relevant for AMP case, training was fine for Float32 case.

Attention mask starts to include -inf values and after dropout returns NaN for -inf.

drop = nn.Dropout()
drop(torch.tensor([-np.inf, 1], dtype=torch.float))

If attention has -inf then matmul between -inf and zero return NaN

How do I deal with those -inf values? Simply excluding or replacing them is not possible because it’s not a math operation.

Context: Is it right use torch.nan_to_num() after each operation?


Why example for torch amp show incorrect results?

Question:
Hello everyone!
I have rtx 3080 and I would like to train Bert.
But when I start example for amp from Automatic Mixed Precision — PyTorch Tutorials 1.9.1+cu102 documentation I get that without amp model trains faster than with amp.
Can someone help me understand why?
For large batch size amp is faster, but it use more memory of card.
net = make_model(in_size, out_size, num_layers)
opt = torch.optim.SGD(net.parameters(), lr=0.001)

start_timer()
for epoch in range(epochs):
    for input, target in zip(data, targets):
        output = net(input)
        loss = loss_fn(output, target)
        loss.backward()
        opt.step()
        opt.zero_grad() # set_to_none=True here can modestly improve performance
end_timer_and_print("Default precision:")

Default precision:
Total execution time = 1.803 sec
Max memory used by tensors = 826428416 bytes

use_amp = True

net = make_model(in_size, out_size, num_layers)
opt = torch.optim.SGD(net.parameters(), lr=0.001)
scaler = torch.cuda.amp.GradScaler(enabled=use_amp)

start_timer()
for epoch in range(epochs):
    for input, target in zip(data, targets):
        with torch.cuda.amp.autocast(enabled=use_amp):
            output = net(input)
            loss = loss_fn(output, target)
        scaler.scale(loss).backward()
        scaler.step(opt)
        scaler.update()
        opt.zero_grad() # set_to_none=True here can modestly improve performance
end_timer_and_print("Mixed precision:")

Mixed precision:
Total execution time = 2.268 sec
Max memory used by tensors = 925018112 bytes

def make_model(in_size, out_size, num_layers):
    layers = []
    for _ in range(num_layers - 1):
        layers.append(torch.nn.Linear(in_size, in_size))
        layers.append(torch.nn.ReLU())
    layers.append(torch.nn.Linear(in_size, out_size))
    return torch.nn.Sequential(*tuple(layers)).cuda()

batch_size = 128 # Try, for example, 128, 256, 513.
in_size = 4096
out_size = 4096
num_layers = 3
num_batches = 50
epochs = 10

# Creates data in default precision.
# The same data is used for both default and mixed precision trials below.
# You don't need to manually change inputs' dtype when enabling mixed precision.
data = [torch.randn(batch_size, in_size, device="cuda") for _ in range(num_batches)]
targets = [torch.randn(batch_size, out_size, device="cuda") for _ in range(num_batches)]

loss_fn = torch.nn.MSELoss().cuda()

I also wanted to ask. Is there any way to reduce the memory consumption for the BERT model?
For model I use checkpoint from utils and accumulation step for backward.
I can use only batch size = 32 and have 4-5 iteration per second.
I think rtx 3080 is slow card for deep learning(((

Context: I don’t know how you’ve checked the memory usage, but the max. peak memory could be higher in case kernels are profiled internally, which might require more workspace.


PyTorch 1.9.0. cause RuntimeError: expected scalar type Half but found Float While PyTorch 1.7.1 does not cause any problem

Question:
Hello
I am using Google Colab for training contrastive learning model. All the code is from this notebook

      colab.research.google.com

Google Colaboratory 2

When I use PyTorch 1.9.0+cu102 and Torchvision 0.10.0+cu102 I got this error (RuntimeError: expected scalar type Half but found Float ) BUT when I use PyTorch 1.7.1 and Torchvision 0.8.2 everything is fine. Thanks for the help
#################################################################################
RuntimeError                              Traceback (most recent call last)
 in ()
376     print(vars(args))
377
 → 378     main(args)
379
380
3 frames
 in main(args)
356     # training loop
357     for epoch in range(epoch_start, args.epochs + 1):
 → 358         train_loss = train(model, train_loader, optimizer, epoch, args)
359         results[‘train_loss’].append(train_loss)
360         #test_acc_1 = test(model.encoder_q, memory_loader, test_loader, epoch, args)
 in train(net, data_loader, train_optimizer, epoch, args)
170             if args.grad_accum==True:
171                 if ((i + 1) % args.accum_iter == 0) or (i + 1 == len(data_loader)):
 → 172                     scaler.scale(loss).backward()
173                     scaler.step(train_optimizer)
174                     scaler.update()
/usr/local/lib/python3.7/dist-packages/torch/_tensor.py in backward(self, gradient, retain_graph, create_graph, inputs)
253                 create_graph=create_graph,
254                 inputs=inputs)
 → 255         torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
256
257     def register_hook(self, hook):
/usr/local/lib/python3.7/dist-packages/torch/autograd/init.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)
147     Variable.execution_engine.run_backward(
148         tensors, grad_tensors, retain_graph, create_graph, inputs,
 → 149         allow_unreachable=True, accumulate_grad=True)  # allow_unreachable flag
150
151
RuntimeError: expected scalar type Half but found Float
#################################################################################

The loss  and the training loop are as the following:
def contrastive_loss(self, im_q, im_k):
    # compute query features

    q = self.encoder_q(im_q)  # queries: NxC

    #print('q.dtype',q.dtype)

    q = nn.functional.normalize(q, dim=1)  # already normalized

    #print('q.dtype',q.dtype)

    # compute key features

    with torch.no_grad():  # no gradient to keys

        # shuffle for making use of BN

        im_k_, idx_unshuffle = self._batch_shuffle_single_gpu(im_k)

        k = self.encoder_k(im_k_)  # keys: NxC

        #print('k.dtype',k.dtype)

        k = nn.functional.normalize(k, dim=1)  # already normalized

        #print('k.dtype',k.dtype)

        # undo shuffle

        k = self._batch_unshuffle_single_gpu(k, idx_unshuffle)

        #print('k.dtype',k.dtype)

    # compute logits

    # Einstein sum is more intuitive

    # positive logits: Nx1

    l_pos = torch.einsum('nc,nc->n', [q, k]).unsqueeze(-1)

    #print('l_pos.dtype',l_pos.dtype)

    # negative logits: NxK

    l_neg = torch.einsum('nc,ck->nk', [q, self.queue.clone().detach()])

    #print('l_neg.dtype',l_neg.dtype)

    # logits: Nx(1+K)

    logits = torch.cat([l_pos, l_neg], dim=1)

    #print('logits.dtype',logits.dtype)

    # apply temperature

    logits /= self.T

    #print('logits.dtype',logits.dtype)

    # labels: positive key indicators

    labels = torch.zeros(logits.shape[0], dtype=torch.long).cuda() #.to(self.device)

    #print('labels.dtype',labels.dtype)

    ###

    #labels = labels.type(torch.LongTensor).cuda()

    ###

    loss = nn.CrossEntropyLoss().cuda()(logits, labels)            #.to(self.device)(logits, labels)

    #print('loss.dtype',loss.dtype)

    return loss, q, k

def forward(self, im1, im2):

    """

    Input:

        im_q: a batch of query images

        im_k: a batch of key images

    Output:

        loss

    """

    # update the key encoder

    with torch.no_grad():  # no gradient to keys

        self._momentum_update_key_encoder()

    # compute loss

    if self.symmetric:  # asymmetric loss

        loss_12, q1, k2 = self.contrastive_loss(im1, im2)

        loss_21, q2, k1 = self.contrastive_loss(im2, im1)

        loss = loss_12 + loss_21

        print('loss.dtype',loss.dtype)

        k = torch.cat([k1, k2], dim=0)

    else:  # asymmetric loss

        loss, q, k = self.contrastive_loss(im1, im2)

    self._dequeue_and_enqueue(k)

    return loss

train for one epoch
def train(net, data_loader, train_optimizer, epoch, args):
i = 0

scaler = torch.cuda.amp.GradScaler(enabled=args.use_amp)

net.train()

adjust_learning_rate(train_optimizer, epoch, args)

total_loss, total_num, train_bar = 0.0, 0, tqdm(data_loader)

for clip_1, clip_2 in train_bar:

    clip_1 = clip_1.cuda(non_blocking=True)   #.to(args.device)

    clip_2 = clip_2.cuda(non_blocking=True)   #.to(args.device)

    #print('clip_1.shape', clip_1.shape)

    #print('clip_2.shape', clip_2.shape)

    with torch.cuda.amp.autocast(enabled=True):

        loss = net(clip_1, clip_2)

        #print('loss.dtype',loss.dtype)

    if args.grad_accum==True and args.batch_size>1:

        loss = loss / args.accum_iter

    #print('loss',loss)

    #print(loss.shape)

    if args.grad_accum==False:

        train_optimizer.zero_grad()

        #loss.backward()

        #train_optimizer.step()

        scaler.scale(loss).backward()

        scaler.step(train_optimizer)

        scaler.update()

    else:

        if args.grad_accum==True:

            if ((i + 1) % args.accum_iter == 0) or (i + 1 == len(data_loader)):

                scaler.scale(loss).backward()

                scaler.step(train_optimizer)

                scaler.update()

                # zero the parameter gradients

                train_optimizer.zero_grad()

    total_num += data_loader.batch_size

    total_loss += loss.item() * data_loader.batch_size

    train_bar.set_description('Train Epoch: [{}/{}], lr: {:.6f}, Loss: {:.4f}'.format(epoch, args.epochs, train_optimizer.param_groups[0]['lr'], total_loss / total_num))

    i+=1

    if i%30==0:

         torch.save({'epoch': epoch, 'state_dict': net.state_dict(), 'optimizer' : train_optimizer.state_dict(),}, args.results_dir + '/model_last.pth')

return total_loss / total_num

Context:
Thanks for the notebook!
I can reproduce the issue in Colab using the older 1.9.0 release, while 1.9.1 works fine (also locally).
I’ve used
!pip install torch==1.9.1+cu111 https://download.pytorch.org/whl/torch_stable.html

in the first cell to update PyTorch.


Question:
What are the downsides to using both mixed precision packages together? Currently, my script is something like this to choose one or the other:
with autocast(enabled=device.type == 'cuda' and apex is None):
	logps = model(images)
	loss = criterion(logps, labels)
	if apex is None:
		scaler.scale(loss).backward()
		scaler.step(optimizer)
		scaler.update()
	else:
		with amp.scale_loss(loss, optimizer) as scaled_loss: # type: torch.FloatTensor
			scaled_loss.backward()
		optimizer.step()

Is this conditional split necessary? Is there a specific reason not to mix them together, or is it relatively benign? It would make my code cleaner to remove the apex is None checks if unneeded but I’m not an expert on mixed precision so I don’t know the potential issues this may cause. I am considering something like this:
with autocast(enabled=device.type == 'cuda'):
	logps = model(images)
	loss = criterion(logps, labels)
	with amp.scale_loss(loss, optimizer) as scaled_loss:
		scaler.scale(scaled_loss).backward()
	scaler.step(optimizer)
	scaler.update()

Context: I’m not sure why you would like to mix them, as torch.cuda.amp is also providing the GradScaler class so there is no need to use the deprecated apex.amp one.


Explanation of exact effect of AMP on memory usage

Question:
I’m writing a blog post breaking down Pytorch memory usage at each step of training, including special cases like training with mixed precision. I have come up with an accurate memory model but it fails in the case of AMP. Here is a link to a Colab notebook with the memory model, where you can replicate what I describe in this post.
From what I understand of how mixed precision works, it should save you exactly half of the memory that the forward pass would normally take (see this and this 1 and this - this last one uses apex, not the native pytorch amp, but is the first empirical evidence I have seen that amp decreases forward pass memory by half). This is because the forward activations are stored in fp16, although the gradients and model weights are in fp32. However in experiments that I have run, I have seen erratic memory usage for AMP. I would like to understand what is going on, so that I can consistently get the memory savings that I expect.
I have written a code to estimate maximum memory usage based on the model size, the input size, the batch size, the optimizer type, and whether AMP is being used. The first function estimates the memory and the second function runs the code to measure the actual memory usage. The estimator function is accurate to the true memory usage, except when use_amp=True.
One note: There are two different measures of memory usage that are relevant here, and they don’t always line up with each other. 1: The memory reported by torch.cuda.memory_allocated. 2: The memory shown by nvidia-smi after the run.
Here are six runs with slightly different parameters, with different outcomes, and my observations.
Run 1:
Function: test_memory(batch_size=64, use_amp=False)
Estimated max memory: 38121984 (this includes the model, gradients, optimizer momentum values, and forward pass).
True max torch.cuda.memory_allocated: 38121984 (exactly correct)
nvidia-smi memory after run: 501MB
Observation: the prediction works as expected without use_amp
Run 2:
Function: test_memory(batch_size=64, use_amp=True)
Estimated max memory: 35536128 (reducing forward pass memory estimate by 1/2).
True max torch.cuda.memory_allocated: 39634432
nvidia-smi memory after run: 503MB
Observation: max allocated memory is greater than without use_amp. This is the opposite of what I would expect.
Run 3:
Function: test_memory(batch_size=128000, use_amp=False)
Estimated max memory: 10375350784
True max torch.cuda.memory_allocated: 10375350784 (exactly correct)
nvidia-smi memory after run: 10667MiB / 11441MiB
Observation: the prediction works as expected without use_amp. The only difference is that with this much larger batch, according to nvidia-smi, CUDA appears to cache much more data.
Run 4:
Function: test_memory(batch_size=140000, use_amp=True)
Estimated max memory: 5751884032
True max torch.cuda.memory_allocated: 5723155456 (very similar to estimation which uses 1/2 multiplier for forward pass memory).
nvidia-smi memory after run: 11429MiB / 11441MiB
Observation: torch.cuda.memory_allocated is pretty close to the prediction. But the memory reflected in nvidia-smi is about double the memory shown by torch.cuda.memory_allocated.
Run 5:
Function: test_memory(batch_size=140000, use_amp=False)
Estimated max memory: 11470817792 (more than the available memory).
True max torch.cuda.memory_allocated: [Memory Error]
nvidia-smi memory after run: 11427MiB / 11441MiB
Observation: as expected, the GPU runs out of memory. When compared with run 4, this shows that use_amp does save a small amount of memory.
Run 6:
Function: test_memory(batch_size=150000, use_amp=True)
Estimated max memory: [Memory Error]
True max torch.cuda.memory_allocated: [Memory Error]
nvidia-smi memory after run: 11427MiB / 11441MiB
Observation: Practically, we don’t see the memory savings that AMP should enable - with 50% memory savings, we should be able to double the non-AMP batch size from run 3 to 256000.
The erratic behavior can be summarized as follows:
1 - For small batches without AMP, it looks like very little memory is cached, but for large batches without AMP, almost all memory is cached (based on nvidia-smi run 1 vs run 3).
2 - For small batches with AMP, torch.cuda.memory_allocated shows an increase in memory usage, but for large batches with AMP, torch.cuda.memory_allocated shows the expected memory savings of 1/2 the forward pass (based on torch.cuda.memory_allocated run 2 vs run 4).
3 - For large batches, torch.cuda.memory_allocated shows the expected memory savings of 1/2 but nvidia-smi shows no memory savings (based on run 4 torch.cuda.memory_allocated and nvidia-smi). My theory of why this happens is that the forward pass is first computed in fp32, then cast to fp16, resulting in a large amount of cached memory. I looked at the source code for autocast and it was a bit opaque - I don’t see where it explicitly casts to fp16.
All of these results can be replicated with the shared Colab notebook.
How should I expect AMP to affect memory usage? Does CUDA caching restrict available memory? I’m particularly curious about run 4, where torch.cuda.memory_allocated shows the expected AMP memory savings, but nvidia-smi shows no savings and I get a memory error. Most importantly, is it possible in PyTorch to save the expected 50% of memory on the forward pass when using AMP, like you can in Apex?
@ptrblck calling in the big guns on this one

Context: @ptrblck do you have any insight on why I am unable to achieve the expected 50% memory savings with AMP? Or a resource you could point me to where I can read up on why AMP doesn’t provide the expected memory savings in pytorch?


When I use amp for accelarate the model, i met the problem“RuntimeError: CUDA error: device-side assert triggered”?

Question:
I wangt to use amp for mixed-precision training. But when i set amp.autocast is true:

image889×427 42.3 KB

I met the problem as follows:

image950×89 7.54 KB

What puzzles me is i can train the model normally, when i set the amp.autocast is false. So how can i to solve the problem?

Context: Could you rerun the script via CUDA_LAUNCH_BLOCKING=1 python script.py args and post the stadcktrace here by wrapping it into three backticks ```, please?


2x faster training than default precision when model is just 3 linear layer, but when the model is just 1 conv layer, mixed precision training is slower than default precision

Question:
OS: ubuntu 16.04
Cuda: 10.2
pytorch: 1.8.0
I ran the demo https://pytorch.org/tutorials/recipes/recipes/amp_recipe.html#a-simple-network 1.
default precision training time is 1.760 sec and mixed precision training time is 0.833. 2x speed up. But when I use torch.nn.Conv2d, amp training is slower than default model. please tell me why?

Context:
Thanks for the update!
I’ve added some missing warmup iterations, since you are using cuDNN’s benchmark mode, so the first iteration will see a slowdown due to the internal profiling.
I can reproduce the slower kernel execution time in FP16 using the default memory format, but see a speedup using memory_format=torch.channels_last, which will avoid the internal permutations via:
cudnn::ops::nchwToNhwcKernel
cudnn::ops::nhwcToNchwKernel

Used code:
import torch
import torch.nn as nn
import time

torch.backends.cudnn.benchmark = True

conv = nn.Conv2d(3, 64, kernel_size=3, stride=2, padding=1, bias=False).cuda()
x = torch.randn(64, 3, 224, 224).cuda()

# warmup
for _ in range(10):
    out = conv(x)
    out.backward(torch.rand_like(out))
grad = torch.rand_like(out)

nb_iters = 100
torch.cuda.synchronize()
t0 = time.perf_counter()
for _ in range(nb_iters):
    out = conv(x)
    out.backward(grad)
torch.cuda.synchronize()
t1 = time.perf_counter()
print('{} s/iter'.format((t1 - t0)/nb_iters))

# warmup
conv.to(memory_format=torch.channels_last)
x = x.to(memory_format=torch.channels_last)
for _ in range(10):
    with torch.cuda.amp.autocast():
        out = conv(x)
        out.backward(torch.rand_like(out))
grad = torch.rand_like(out)

torch.cuda.synchronize()
t0 = time.perf_counter()
for _ in range(nb_iters):
    with torch.cuda.amp.autocast():
        out = conv(x)
        out.backward(grad)
torch.cuda.synchronize()
t1 = time.perf_counter()
print('{} s/iter'.format((t1 - t0)/nb_iters))

Default memory format:
0.0012196038290858268 s/iter
0.0014858345501124858 s/iter

Channels-last:
0.0012191115505993367 s/iter
0.0008443104848265648 s/iter


Question: image1445×1276 115 KB

Context: CUDA operations are executed asynchronously, so you would have to synchronize the code before starting and stopping the timer. The torch.utils.benchmark utility provides this functionality as well as e.g. warmup iterations.


Using Pytorch’s AMP with multiple scaler backwards per epoch

Question:
I’m trying to implement Wasserstein-GP with Pytorchs’s Automatic Mixed Precision. In Wasserstein GAN the critic networks gets updated more frequently than generator (e.g. 5 times per each iteration).
My code goes something like:
for i in range(5):
   with torch.cuda.amp.autocast():
     discriminator_optimizer.zero_grad()
         # do some stuff
      loss_discriminator = compute_discriminator_loss()
      loss_gradient = compute_gradient_penalty_loss()
      ld = loss_gradient +  loss_discriminator
   scaler.scale(ld).backward()
   scaler.step(discriminator_optimizer)
with torch.cuda.amp.autocast():
   generator_optimizer.zero_grad()
   # do some stuff
   loss_generator = compute_generator_loss()
scaler.scale(loss_generator ).backward()
scaler.step(generator_optimizer)
scaler.update()

but I get the error
RuntimeError: step() has already been called since the last update().

When I call the backward() on discriminator loss for the second time.
The error does not appear if I update the scaler after each backward() call, but the Pytorchs examples states that " scaler.update should only be called once".
So what’s the correct way to update the scaler here?
(I’m using pytorch 1.9.0)

Context:
hi,
not an expert on amp, but it seems you cant repeat step() with the same scale factor. it has to be updated. i think it is by design because it makes sens that way. the scale factor is designed to follow the dynamics of gradients which change through step(). so, every time you step(), you need to update(). you cant step several times without update, because you will leave the scale factor behind.
i think you should use 2 gradient scalers because your both losses are progressing at different rates. so, each one needs its own scale factor that is updated using update. this will allow you to update the scale factor of the discriminator as often as you like. it does not seem that you retaining any gradient to share between disc and gen, so, you can scale their gradient separately.
i think the pytorch example works because both losses advance with the same rate.


Question:
Hello,
I want to use AMP on a ResNet-18 which was trained without AMP (plain Float32) on CIFAR-10.
However, when I wrap the forward pass of the model in a torch.cuda.amp.autocast() block, the output of the network gets nan. When I deactivate AMP with torch.cuda.amp.autocast(enabled=False) I get the expected output values.
Below I attached a minimal example that reproduces my problem.
Does anyone have an idea why the output is always nan?
import torch

from models.cifar10_models.resnet import ResNet18

resnet = ResNet18(num_classes=10)
resnet.load_state_dict(torch.load('./pretrained_models/resnet18_cifar10.pth'))

inputs = torch.randn((124, 3, 32, 32), requires_grad=True).cuda()

resnet.cuda()
resnet.eval()
for i in range(500):
    with torch.cuda.amp.autocast():
        outputs = resnet(inputs)

print(outputs)

Edit:
I have noticed that reproducing the problem is difficult when I am loading a local model. Therefore, here another example with a pretrained ResNet-18 on ImageNet. Running the code below also produces nan as output:
import torch
from torchvision.models.resnet import resnet18

resnet = resnet18(pretrained=True)
resnet.fc = torch.nn.Linear(512, 10)

inputs = torch.randn((124, 3, 32, 32), requires_grad=False).cuda()

resnet.cuda()
resnet.eval()
with torch.cuda.amp.autocast(enabled=True):
    outputs = resnet(inputs)

print(outputs)

Context:
Unfortunately I cannot reproduce the issue.
No matter how often I try I never get nan
This might be a version problem.
It tried it on:
CUDA 10.2 / Pytorch1.8.1
as well as on:
CUDA 10.2 / Pytorch1.9.0
What are you using?


Models can’t seem to learn with FP16 training and time taken for training is absurd

Question:
I’m trying to benchmark the performance (time taken to train, GPU memory usage) while training models in FP16(“O3”) and FP32(“O0”).
It is known that FP16 takes a comparatively low GPU memory and trains faster in Tensor Core
I Used Tesla T4 from (Google Colab) to train these models which from what i read from internet, supports Tensor Core.
I trained the model in all 4 amp opt levels “O0,1,2,3”.

But I couldn’t see any significant difference in training time, and for the most part model doesn’t learn anything.
I don’t know why those memory values happen to be that.
I can see the  gradient vanish to NaN after the first step. I tried adding BatchNorm, no use.

I’m not sure what am I doing wrong or missing.
Any help would be great, thanks in advance.
wandb output link : Weights & Biases
stdout are attached in each run’s page
My Model
class Net(nn.Module):

    def __init__(self):
        super(Net,self).__init__()

        self.convs = nn.Sequential(
            nn.Conv2d(3, 32, 3, 1),
            nn.ReLU(),
            nn.BatchNorm2d(32),
            nn.Conv2d(32, 64, 3, 1),
            nn.ReLU(),
            nn.BatchNorm2d(64),
            nn.Conv2d(64, 128, 3, 1),
            nn.ReLU(),
            nn.BatchNorm2d(128),
            nn.Conv2d(128, 256, 3, 1),
            nn.ReLU(),
            nn.BatchNorm2d(256),
            nn.MaxPool2d(2),
            nn.Dropout(0.25)
        )

        self.linears = nn.Sequential(
            nn.Linear(12*12*256,128),
            nn.ReLU(),
            nn.Dropout(0.5),
            nn.Linear(128,10)
        )

    def forward(self,x):
        x = self.convs(x)
        x = x.view(x.shape[0],-1)
        x = self.linears(x)
        logits = F.log_softmax(x,dim=1)
        return logits

My Training Loop
class Pipeline:
    def __init__(self,config):
         # do stuff with config
        # optimizer -> Adam
        self.loss_fn = nn.NLLLoss()

    def trainStep(self,batch,half:bool=False):
        X,y = batch
        X,y = X.to(self.device),y.to(self.device)

        # forward
        self.model.train()
        logits = self.model(X)
        loss = self.loss_fn(logits,y)
        acc = torch.argmax(logits.detach(),dim=1)==y
        acc = torch.mean(acc.float())

        # backward
        with apex.amp.scale_loss(loss, self.optimizer) as scaled_loss:
            scaled_loss.backward()
        self.optimizer.step()

        return loss,acc

    def trainLoop(self,epochs=None,half:bool=False,opt:str="O0",log_config=None):

        if epochs is None:
            epochs = self.epochs

        self.model,self.optimizer = apex.amp.initialize(self.model,self.optimizer,opt_level=opt,loss_scale="dynamic")
        wandb.watch(self.model,self.loss_fn,log="all",log_freq=1)

        eg_count = 0
        t = trange(epochs)
        for epoch in t:
            b = tqdm(self.train_loader)
            for batch in b:
                eg_count += len(batch)
                loss,acc = self.trainStep(batch)

                self.trainLog(loss,acc,eg_count,epoch)

        torch.cuda.empty_cache()

    def pipe(self,project:str,run:str,half:bool=False,val:bool=False,test:bool=False):
        with wandb.init(project=project,name=run,config=self.config):
            opt = run[-2:]

            self.trainLoop(opt=opt)

        # create new model instance after for next run
        self.model = self.config['modelClass']().to(self.device)
        self.optimizer = torch.optim.Adam(self.model.parameters(),lr=self.learning_rate)

config = {
    "epochs" : 5,
    "batch_size" : 1024,
    "learning_rate" : 1e-2,
    "half" : False,
    "classes" : 10,
    "dataset" : "CIFAR10",
    "architecture" : "CNN",
    "kernels" : [32,64,128,256],
    "modelClass" : Net
}

pl = Pipeline(config)
pl.pipe("first",run="fp32-O0")
pl.pipe("first",run="fp16-O3")
pl.pipe("first",run="fp-Mixed-O2")
pl.pipe("first",run="fp-Mixed-O1")

Context: apex.amp is deprecated and we thus recommend to use the native mixed-precision training util. via torch.cuda.amp as described here 8.