Does amp training can use torch.nn.DataParallel at the sametime？

Question:
I try to use amp with pytorch1.6 to speed up my training code.
But I have a problem ,when I use nn.DataParallel.
I print some Intermediate variable.
I find the tensor is float16 in one gpu, but float32  in two gpus.
Is it support DataParallel model to use mixed-precision training?
in one gpu:

fake_image_orig:  torch.float16
gen loss:  torch.float32
discriminator_out dtype:  torch.float16
pred_fake:  torch.float16
amp discriminor
discriminator_out dtype:  torch.float16
self.get_zero_tensor(input) dtype:  torch.float16
input dtype:  torch.float16
self.get_zero_tensor(input) dtype:  torch.float16

two gpus:

discriminator_out dtype:  torch.float32
self.get_zero_tensor(input) dtype:  torch.float32
input dtype:  torch.float32
self.get_zero_tensor(input) dtype:  torch.float32
input dtype:  torch.float32
discriminator_out dtype:  torch.float32
self.get_zero_tensor(input) dtype:  torch.float32
input dtype:  torch.float32
self.get_zero_tensor(input) dtype:  torch.float32
input dtype:  torch.float32
fake_image_orig:  torch.float32
gen loss:  torch.float32
discriminator_out dtype:  torch.float32
pred_fake:  torch.float32
fake_image_orig:  torch.float32
gen loss:  torch.float32
discriminator_out dtype:  torch.float32
pred_fake:  torch.float32
fake_image_orig:  torch.float32
gen loss:  torch.float32
fake_image_orig:  torch.float32
gen loss:  torch.float32
discriminator_out dtype:  torch.float32

Context:
Yes, amp is usable in nn.DataParallel as described here 74.
I guess you might have missed this note about the @autocast() decorator for the forward method, if you are using nn.DataParallel.


What is the correct way of computing a grad penalty using AMP?

Question:
I’m converting my distributed code to work with PyTorch’s AMP, but I have confusion on how I should compute my grad penalty as part of my WGAN-GP loss. I have referred to the documentation 10 on this matter but I believe my use case is slightly different.
Here’s my code on computing the grad penalty:
out = discrim(sample)

gradients = torch.autograd.grad(inputs=sample, outputs=out,
    grad_outputs=torch.ones(out.shape).to(sample.device),
    create_graph=True, retain_graph=True, only_inputs=True)[0]

You may assume that sample is a tensor yielded by my generator and out is a N x 1 tensor. I’ll start off with saying that this code works without amp, but I’m just confused on how I’m suppose to use amp with this. This is what I’m currently trying:
with autocast(False):
    sample = scaler.scale(sample)
    out = discrim(sample)

    gradients = torch.autograd.grad(inputs=sample, outputs=out,
        grad_outputs=torch.ones(out.shape).to(sample.device),
        create_graph=True, retain_graph=True, only_inputs=True)[0]
    gradients = gradients / scaler.get_scale()

Note that sample was computed under autocast(True), but I’m not sure if this matters. This seems to work, but I think I want to compute out under the mixed precision context to be faster. This seems to be slow atm.
This my second attempt at the problem:
with autocast(True):
    out = discrim(x=sample)

    with autocast(False):
        sample = scaler.scale(sample)
        out = scaler.scale(out)

        gradients = torch.autograd.grad(inputs=sample, outputs=out,
            grad_outputs=torch.ones(out.shape).to(sample.device),
            create_graph=True, retain_graph=True, only_inputs=True)[0]

        gradients = gradients / scaler.get_scale()

But I get this error on autograd: One of the differentiated Tensors appears to not have been used in the graph. I’m not really sure on what’s the proper way of doing this. Any help is greatly appreciated!

Context:
I don’t believe your use case is different from what’s shown in the docs 28, aside from more explicit kwargs and outputs being a Tensor with (presumably) more than one element.  Perhaps, instead of

To implement a gradient penalty with gradient scaling, the loss passed to  torch.autograd.grad()  9 should be scaled.

the docs should say

To implement a gradient penalty with gradient scaling, the outputs Tensor(s) passed to  torch.autograd.grad()  9 should be scaled.

Try
with autocast():
    out = discrim(sample)

gradients = torch.autograd.grad(outputs=scaler.scale(out), inputs=sample,
    grad_outputs=torch.ones(out.shape, device=sample.device),
    create_graph=True, retain_graph=True, only_inputs=True)[0]

# proceed as shown in the docs

# if gradients is a Tensor
gradients = gradients / scaler.get_scale()
# if gradients is a tuple
inv_scale = 1./scaler.get_scale()
gradients = [p * inv_scale for p in gradients]

with autocast():
    # compute penalty term from gradients
    # add to loss if needed

scaler.scale(<loss or penalty>).backward()

If that works I will update docs as described above.
FYI in grad, retain_graph defaults to the value of create_graph and only_inputs defaults to True so these kwargs are not needed.  Also, I changed
torch.ones(out.shape).to(sample.device) to torch.ones(out.shape, device=sample.device) which synthesizes grad_outputs directly on the device, avoids an expensive CPU->GPU copy, and is good practice regardless of Amp.


Question:
Hello,
I trained frcnn model with automatic mixed precision and exported it to ONNX. I wonder however how would inference look like programmaticaly to leverage the speed up of mixed precision model, since pytorch uses with autocast():, and I can’t come with an idea how to put it in the inference engine, like onnxruntime.
My specs:
torch==1.6.0+cu101
torchvision==0.7.0+cu101
onnx==1.7.0
onnxruntime-gpu==1.4.0
Model exports just fine:
torch.onnx.export(model,
                  x,
                  "model_16.onnx",
                  verbose=True, do_constant_folding=True, opset_version=12,
                  input_names=input_names, output_names=output_names)

But I wonder how to leverage mixed precision speed up here:

import onnxruntime as ort
ort_session = ort.InferenceSession('model_16.onnx')
outputs = ort_session.run(None, {'input': x.numpy()})

Context: I’m not exactly sure how ONNX exports the model, but if tracing is used, the mixed-precision operations might have been already recorded. Do you see any FP16 operations, if you profile the ONNX model?


Question:
Without AMP
import torch
from torch.cuda.amp import autocast, GradScaler
from torchvision import models

model = models.mobilenet_v2(pretrained=True).cuda()
loss_fnc = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.0001)

X = torch.randn((32,3,300,300), dtype=torch.float32).cuda()
y = torch.randint(0, 1000, (32,), dtype=torch.long).cuda()

model.train()
for j in range(30):
    optimizer.zero_grad()

    y_hat = model(X)
    loss = loss_fnc(y_hat, y)
    loss.backward()
    optimizer.step()
    print (loss.item())

Output:
8.039933204650879
5.690041542053223
3.4787116050720215
1.607206106185913
0.6231755614280701
0.23825135827064514
0.08544095605611801
0.04335329309105873
0.016259444877505302
0.01174827478826046
0.0069425650872290134
0.004459714516997337
0.003734807949513197
0.0024659112095832825
0.0027059323620051146
........................

But with AMP it gives nan’s
model = models.mobilenet_v2(pretrained=True).cuda()
loss_fnc = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.0001)

X = torch.randn((32,3,300,300), dtype=torch.float32).cuda()
y = torch.randint(0, 1000, (32,), dtype=torch.long).cuda()

scaler = GradScaler()
model.train()
for j in range(30):
    optimizer.zero_grad()

    with autocast():
        y_hat = model(X)
        loss = loss_fnc(y_hat, y)

    scaler.scale(loss).backward()
    scaler.step(optimizer)
    scaler.update()

    print (loss.item())

Output:
8.393239974975586
nan
nan
nan
nan
nan
nan
nan
nan
nan
nan
nan
nan
nan
nan
.......................

Context:
It works fine with CUDA 10.2.
I will upgrade my env.
Thanks


Question:
Hi, I have large evaluation data set, which is the same size as the training data set and I’m performing the validation phase during training to be able to control the behavior of the training process.
I’ve added automatic mixed precision in the training phase, but is it safe to wrap the validation step in the training process with amp.autocast() to speed up that forward propagation?
In general, is it safe/recommended to use mixed precision in model evaluation during the tuning process and if it is, what is the right way to implement?
for epoch in range(epochs):
     # Training phase
     train_loss, train_score = self.train_model(trainset)

     # Validation phase
     valid_loss, valid_score = self.valid_model(validset)

@torch.no_grad()
def valid_model(self, dataloader):
     self.eval()

     for batch in tqdm(dataloader):
          # Evaluate with mixed precision
          if self.setting.mixed_precision:

               # Runs the forward pass with autocasting, including loss and score calculation
               with amp.autocast():
                    loss, score = self.validation_step(batch)

Context: Yes, you can also use autocast during the validation step and wouldn’t need to apply the gradient scaling, since no gradient are calculated in this phase.


Question: How to add my project to the ecosystem，what conditions need to be met before filling out the table (Join | PyTorch 3)

Context: As described in the linked page, if you’ve answered “yes” to both questions you should fill out the form to send the application.


Unconverging loss for a simple regression task

Question:
I am working on a project to predict soccer player values from a set of inputs. The data consists of about 19,000 rows and 8 columns (7 columns for input and 1 column for the target) all of numerical values. Here is a glimpse:

head1056×276 30 KB

I am using a fully connected Neural Network for the prediction but the problem is the loss is not decreasing as it should. This is the result I get:

result428×831 62.1 KB

This is how I am creating the dataset and the dataloaders:

dataset566×924 57.6 KB

This is the code I am using to run the model:

gradient964×540 74.1 KB

model1115×809 89.6 KB

I tried changing the learning rate but it is still bad.
What could be the reason behind this? Thank you!

Context: Is ValueEUR the target variable? It seems to have values in 10e7 . Thus, the loss can be pretty big if it is used directly (without scaling).


Preprocessing Dataloader Batches prior to sending them through to model

Question:
Hi there, (first time posting)
I am currently working on a university project where I’m aiming to use two different models on a dataset and combine the results during training. My question is regarding the preprocessing of my datasets. Currently my dataset is a variety of images.
For the first model, each image is converted to a numpy array that represents the information for each colour channel.
For the second model, facial elements of the image are extracted before being passed on to the model.
As you can see, each model requires different preprocessing before the data can be passed to each model. My current approach is to load the entire image dataset into a Dataloader and then while iterating through each batch, perform the preprocessing on the batches before passing them on to the models. I wanted to see if this is the correct approach.
Please let me know what you think. Thanks

Context:
For anyone who finds this and is interested:
My previous understanding was that getitem returned the data + the class. However, after playing around I realise we can return a list of data. My current approach will be to send a list, from getitem where the first element in the array will be the data related to my first model, and the second item in the array will be the corresponding data related to the second model.


The size of a tensor a (1000) must match the size of a tensor b (100) at non-singleton dimension

Question:
I’m a bit at a loss of why I am getting the following error when I run the following model:
class TestModel(torch.nn.Module):

    def __init__(self):
        super(TestModel, self).__init__()

        self.layer1 = torch.nn.Linear(1000, 100)
        self.relu = torch.nn.ReLU()
        self.layer2 = torch.nn.Linear(100,1)

    def forward(self,x):
        x - self.layer1(x)
        x = self.relu(x)
        x = self.layer2(x)
        return x

I get the error when I run the following line:
model = TestModel()
x = torch.randn(5,1000)
model(x)

And yet… when I test it through only the nn.linear function, I get no error, such as the following code:
linear = torch.nn.Linear(1000,100)
linear(x)

I have been staring at the monitor and scratching my head at this. Other similar errors I found online had to do with people using different shapes input vs target in the MSE loss, which is not the case here.
The error also goes away when I change layer1 to:
        self.layer1 = torch.nn.Linear(1000, 1000)

But I don’t want the output of the linear layer to be the same as the input. In fact, the size of the output units should not matter when it comes to the size of the input data (as we should be multiplying a 5x1000 matrix with a 1000x100 one).

Context:
alex_rachkov:

x - self.layer1(x)

You have a typo here.


Question:
My work with Pytorch -
Trained on custom dataset and model.
Github link - project is still not finished !

      GitHub

GitHub - Samuel-Bachorik/PyTorch-AI-road-detection-concurrent_training:... 23

  Artificial intelligence model, training loop, model loader, inference - GitHub - Samuel-Bachorik/PyTorch-AI-road-detection-concurrent_training: Artificial intelligence model, training loop, model l...

Test, new neural network model on custom dataset, Encoder-decoder 16xConv2D layers, Relu Activation

Context:
Amazing work!! I have a few questions:

Does it work in real-time?
Can this model generalize well to sky images? I am guessing it will but wanna know what are your thoughts on it.
How long did it took to train and which GPU use used?


Question:
Hi, im building a jetbot with the sparkfun jetson nano 2GB kit. The V01-00 image worked though the training model isn’t working properly. It seems to have issues with the cuda system. These our the errors I have been getting:
1.RuntimeError: CUDA error: device-side assert triggered - at one of the times I tried to run the program.
2. RuntimeError: cuda runtime error (59) : device-side assert triggered at /media/nvidia/WD_BLUE_2.5_1TB/pytorch-v1.1.0/aten/src/THC/generic/THCTensorMath.cu:16.
3.RuntimeError: cuda runtime error (59) : device-side assert triggered at /media/nvidia/WD_BLUE_2.5_1TB/pytorch-v1.1.0/aten/src/THC/generic/THCTensorMath.cu:26
This is my code:
NUM_EPOCHS = 30
BEST_MODEL_PATH = ‘best_model.pth’
best_accuracy = 0.0
optimizer = optim.SGD(model.parameters(), lr=0.001, momentum=0.9)
for i, data in enumerate(all_dataloader):
for epoch in range(NUM_EPOCHS):

    for images, labels in iter(train_loader):
        images = images.to(device)
        labels = labels.to(device)
        optimizer.zero_grad()
        outputs = model(images)
        loss = F.cross_entropy(outputs, labels)
        loss.backward()
        optimizer.step()

    test_error_count = 0.0
    for images, labels in iter(test_loader):
        images = images.to(device)
        labels = labels.to(device)
        outputs = model(images)
        test_error_count += float(torch.sum(torch.abs(labels - outputs.argmax(1))))

    test_accuracy = 1.0 - float(test_error_count) / float(len(test_dataset))
    print('%d: %f' % (epoch, test_accuracy))
    if test_accuracy > best_accuracy:
        torch.save(model.state_dict(), BEST_MODEL_PATH)
        best_accuracy = test_accuracy

This is the error:

RuntimeError                              Traceback (most recent call last)
 in
13         outputs = model(images)
14         loss = F.cross_entropy(outputs, labels)
—> 15         loss.backward()
16         optimizer.step()
17
/usr/local/lib/python3.6/dist-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)
105                 products. Defaults to False.
106         “”"
 → 107         torch.autograd.backward(self, gradient, retain_graph, create_graph)
108
109     def register_hook(self, hook):
/usr/local/lib/python3.6/dist-packages/torch/autograd/init.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
91     Variable._execution_engine.run_backward(
92         tensors, grad_tensors, retain_graph, create_graph,
—> 93         allow_unreachable=True)  # allow_unreachable flag
RuntimeError: cuda runtime error (59) : device-side assert triggered at /media/nvidia/WD_BLUE_2.5_1TB/pytorch-v1.1.0/aten/src/THC/generic/THCTensorMath.cu:26
picture is attached.
I would appreciate your assistance to proceed in my project. @albanD
pic_!1267×744 43.2 KB

Context:
Hi,
You can try to run with CUDA_LAUNCH_BLOCKING=1 to get a more accurate error.
Also can you try running the code on the CPU? It might be a problem with the embedded GPU?


How does Frequency Pooling make sense in CNNs that operates fully in the Frequency Domain?

Question:
I am reading this paper, image below,

which uses Frequency Pooling, from this paper, image below.

Question
For simplicity, let me assume there is no sinc interpolation, and the convolution kernels are stored in the frequency domain having the same size of the input (N x N).
My understanding of the frequency pooling in the proposed method is the following:

On the forward-pass, we have that the lower frequencies components of the convolution output (N x N) are shifted to the center of the representation.
Next, the high frequency components outside of a central area (say N/2 x N/2) are discarded.
Then, the remaining frequencies as passed to the next layers ( non-linear activation; convolution; frequency pooling; non-linear activation; and so on)

Here is an illustration, notice low frequencies are assumed to be in the center.

Given that, it seems that the discarded frequencies in the frequency pooling process do not contribute at all to the output. If that is the case, then all the frequencies that are discarded in all pooling layers, up to the last frequency pooling, would not be used for the computation of the output.
Wouldn’t this be equivalent to discarding all those frequencies in the input layer?
I seem to be missing something very important on this.

PS: Is there a specific channel to discuss research with this community?

Context: I haven’t checked the paper in detail, but wouldn’t the steps between the frequency pooling (activation, conv etc.) be able to create activation maps with a wider frequency output, which then could be pooled again?


Question:
Hi everyone!
I am working on a project with histopathological images (called Whole-Slide Image). Each of these images is ~1 GB, so they are really hard to handle.
Particularly I struggle when I use DataLoader(num_worker=N) (where N>1) because PyTorch starts loading and preprocessing (slight data augmentation in our case) multiple batches in RAM and then the RAM fills up fast.
I wanted to know if there are other people working on implementing an alternative DataLoader mechanism that could allow us to have multiple workers working on the same batch.
I would also like to know if you have any suggestions regarding this topic.
Since I never opened a Pytorch PR and since I noticed that worker shutdown/handling is a very delicate matter, do you think I could open a draft and then someone could provide some suggestions and support?

Context: This feature request is tracked here 11 and I’m sure contributions are welcome, so please feel free to post your interest in the issue and code owners will follow up with you.


How to make a dataLoader with reqiures_grad=True from a dasaset and not revert reqiures_grad to false?

Question:
I am trying to create a Dataloader using the built-in DataLoader class from a dataset created using the built-in Dataset class, the problem is that the tensor in the DataLoader reverts to reqiures_grad=False. I think the DataLoader class makes a copy or something, what might be the best way of going around this?.
_data=torch.tensor(newData).float();
lebels=torch.tensor(y_train).float();
data=torch.utils.data.TensorDataset(_data,lebels)
print("data shape: ", _data.shape);
print("Lebels shape : ", lebels.shape);
data=torch.utils.data.TensorDataset(data.requires_grad()
, lebels.requires_grad());
_data[0][0].requires_grad #outputs true
trainDataLoader = DataLoader(dataset=_data, batch_size=15, shuffle=True)
it = iter(trainDataLoader)
first = next(it)
first[0][0].requires_grad #outputs False

Context:
I cannot reproduce the issue using:
_data = torch.randn(100, 1)
lebels = torch.randn(100, 1)
data = torch.utils.data.TensorDataset(_data,lebels)
print("data shape: ", _data.shape);
print("Lebels shape : ", lebels.shape);
data = torch.utils.data.TensorDataset(_data.requires_grad_(), lebels.requires_grad_())

print(_data[0][0].requires_grad)
> True

x, y = data[0]
print(x.requires_grad)
> True
print(y.requires_grad)
> True

trainDataLoader = torch.utils.data.DataLoader(dataset=data, batch_size=15, shuffle=True)
it = iter(trainDataLoader)
first = next(it)
first[0][0].requires_grad
> True

In your current code snippet you are trying to “call” the requires_grad attribute, which is a bool value and which would raise an error:
data=torch.utils.data.TensorDataset(data.requires_grad(), lebels.requires_grad())

so I changed it to the inplace ops: tensor.requires_grad_().
Also, you are passing the _data tensor (not the data Dataset) to the DataLoader, which you also might want to change.


Question:
Hi ,hope all of you are fine.
I am trying out to come up with a viable crypto-price-prediction model, but as it is quite unstable, I am figuring out that maybe the technical indicators like RSI,4 EMA, hash ribbons, etc, which professional traders use will do the trick.
Right now, most of the tutorials and blogs about using LSTMs, GRUs, etc, are just fooling and I couldn’t find any workable solution.
So, if anyone can guide me, will be very grateful.
Thanks.

Context: can you elaborate a bit? ‘coming up with a model’ can mean a million different things.


Torch.nn.functional.cosine_similarity got unexpected behavier with torch.int8 dtype

Question:
PyTorch version: 1.9.0
torch.nn.functional.cosine_similarity got unexpected behavier with torch.int8 dtype.
>>> a = torch.randn(10)*100
>>> a
tensor([ 47.8993, -27.2694,  20.9548, -13.5573, -54.0388, -76.8524,  -7.4037,
        -95.7477, 108.9276, -44.5625])
>>> b = a
>>> F.cosine_similarity(a, b, dim=-1)
tensor(1.)
>>> F.cosine_similarity(a.to(torch.int8), b, dim=-1)
tensor(3.5215e+12)
>>> F.cosine_similarity(a.to(torch.int16), b, dim=-1)
tensor(1.0000)

As we can see, F.cosine_similarity(a.to(torch.int8), b, dim=-1) got result of tensor(3.5215e+12), while the expected result is tensor(1.).

Context: This looks like an overflow using int8 internally. Would you mind creating an issue on GitHub so that we could track and fix it?


Question:
My dataset has temporal data of coordinates of shape (1000,5,2). Each (5,2) is a pair of 5 coordinates that looks like below. 1000 of such arrays are merged together to create my dataset.
array([[109, 106],
[109, 106],
[110, 109],
[110, 109],
[108, 107]])
I am sending it to an LSTM but I am getting the following error while passing the output to the loss calculation. My output layer has 2 classes.
ValueError: Expected input batch_size (5) to match target batch_size (2)
The following is the code that I tried:
#Create Data Loader
input_size = 2
batch_size = 1
hidden_dim = 12
output_size = 2
epochs = 20
use_cuda = torch.cuda.is_available()
params = {'batch_size': batch_size, 'shuffle': False, 'num_workers': 2, 'pin_memory': True} if use_cuda else {}

train_loader = data.DataLoader(train_set, **params)
#test_loader = data.DataLoader(test_set, **params)

#model
class Model(nn.Module):
    def __init__(self, device,input_size, output_size, hidden_dim, n_layers):
        super(Model, self).__init__()

        # Defining some parameters
        self.hidden_dim = hidden_dim
        self.n_layers = n_layers
        self.device = device
        self.softmax = nn.Softmax(dim=-1)

        #Defining the layers
        # LSTM Layer
        self.lstm = nn.LSTM(input_size, hidden_dim, n_layers, batch_first=True)
        # Fully connected layer
        self.fc = nn.Linear(hidden_dim, output_size)

    def forward(self, x):
        batch_size = x.size(0)

        # Initializing hidden state for first input using method defined below
        hidden = self.init_hidden(batch_size).to(device)
        cell_state = self.init_hidden(batch_size).to(device)

        # Passing in the input and hidden state into the model and obtaining outputs
        out, hidden = self.lstm(x, (hidden,cell_state))

        # Reshaping the outputs such that it can be fit into the fully connected layer
        out = out.contiguous().view(-1, self.hidden_dim)
        out = self.fc(out)
        out = self.softmax(out)

        return out, hidden

    def init_hidden(self, batch_size):
        # This method generates the first hidden state of zeros which we'll use in the forward pass
        hidden = torch.zeros(self.n_layers, batch_size, self.hidden_dim)
        return hidden

# Instantiate the model with hyperparameters
model = Model(device,input_size=input_size, output_size= output_size, hidden_dim= hidden_dim, n_layers=2).to(device)

# Define hyperparameters and loss function
lr=0.01
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=lr)

#Train
for epoch in range(epochs):
    sum_loss = 0.0
    total = 0
    for batch_idx, (X, y) in enumerate(train_loader):
        X, y = X.to(device), y.to(device).view(-1, )
        optimizer.zero_grad() # Clears existing gradients from previous epoch
        output, hidden = model(X)
        loss = criterion(output, y.view(-1).long())
        loss.backward() # Does backpropagation and calculates gradients
        optimizer.step() # Updates the weights accordingly
        sum_loss += loss.item()*y.shape[0]
        total += y.shape[0]
        if (batch_idx+1) % 1000 == 0:
            print('Epoch: {}/{}.............'.format(epoch+1,epochs), end=' ')
            print("Loss: {:.4f}".format(sum_loss/total))

Context:
I guess a view or reshape operation might be wrong and in particular this line of code might be interesting to check:
        # Reshaping the outputs such that it can be fit into the fully connected layer
        out = out.contiguous().view(-1, self.hidden_dim)

Here you are pushing all “additional” dimension/shapes to dim0, which is the batch dimension and should stay constant.
Use: out = out.contiguous().view(out.size(0), -1) instead as it would keep the batch size equal and would then flatten the additional values into the feature dimension.
If a shape mismatch is raised in the linear layer afterwards, adapt the in_features to the expected value.
Also, remove the nn.Softmax layers, since nn.CrossEntropyLoss expects raw logits and will apply F.log_softmax internally.


How to properly use Cross Entropy with weights for node classification

Question:
I am programming my first GNN and want to do a node classification. However I feel like my predictions do not get trained properly. I think it has to do with the Cross Entropy Loss.
I want to perform a binary classification on every node in my Graph. I have two classes, 0 and 1. About 75% of the nodes belong to class 0 and 25% to class 1.
This is the Network:
import torch
import torch.nn.functional as F
from torch_geometric.nn import GCNConv

class Network(torch.nn.Module):
    def __init__(self, hidden_channels):
        super(Network, self).__init__()
        self.GCNConv1 = GCNConv(in_channels=6, out_channels = hidden_channels)
        self.GCNConv2 = GCNConv(in_channels=hidden_channels, out_channels = hidden_channels)
        self.GCNConv3 = GCNConv(in_channels = hidden_channels, out_channels = 2)

    def forward(self, x, edge_index, edge_weight):

        edge_weight = 7.91 - edge_weight

        h = self.GCNConv1(x, edge_index, edge_weight)
        h = F.relu(h)
        h = self.GCNConv2(h, edge_index, edge_weight)
        h = F.relu(h)
        h = self.GCNConv3(h, edge_index, edge_weight)
        return h

And the training process looks like this:
preds has shape: preds.shape = [num_nodes,2]
batch.y has shape: batch.y.shape = [num_nodes] , batch.y holds 0 or 1 depending on the class
import torch_geometric

dataset = SimulatinData("")
dataset = dataset.shuffle()
train_set = dataset[:25000]
batch_size = 100

network = Network(hidden_channels = 32).double()
train_loader = torch_geometric.data.DataLoader(train_set, batch_size = batch_size, shuffle = True)
optimizer = torch.optim.Adam(network.parameters(), lr=0.01)

for epoch in range(10):
    for batch in train_loader:

        network.train()
        preds = network(batch.x, batch.edge_index, batch.edge_weights)

        class_weight = torch.tensor([1,4]).double()
        loss = F.binary_cross_entropy_with_logits(preds, F.one_hot(batch.y).double(), weight = class_weight)

        loss.backward() # Calculate Gradients
        optimizer.step() # Update Weights
        optimizer.zero_grad() # Delete Gradients from previous loop

No matter for how many epochs I train my network, the predictions always look like this:
epoch:0
tensor([[ 0.6972, -1.0346],
        [ 0.6972, -1.0346],
        [ 0.6972, -1.0346],
        ...,
        [ 0.6972, -1.0346],
        [ 0.6972, -1.0346],
        [ 0.6972, -1.0346]], dtype=torch.float64, grad_fn=<AddBackward0>)
epoch:1
tensor([[ 0.7204, -0.9803],
        [ 0.7204, -0.9803],
        [ 0.7204, -0.9803],
        ...,
        [ 0.7204, -0.9803],
        [ 0.7204, -0.9803],
        [ 0.7204, -0.9803]], dtype=torch.float64, grad_fn=<AddBackward0>)
epoch:100
tensor([[ 0.9839, -0.7851],
        [ 0.9839, -0.7851],
        [ 0.9839, -0.7851],
        ...,
        [ 0.9839, -0.7851],
        [ 0.9839, -0.7851],
        [ 0.9839, -0.7851]], dtype=torch.float64, grad_fn=<AddBackward0>)

So it just assmues every node belongs to category 0. So the network is “correct” for 75% of the nodes but that is of course not the prediction I would like to have. No matter what class_weight I choose for the cross entropy, I always get these predictions where the GNN just says every node belongs to 0. Do you know what could go wrong in my code?

Context:
The weight argument in nn.BCEWithLogitsLoss and F.binary_cross_entropy_with_logits gives the weight values per sample, not per class. From the docs:

weight (Tensor 2, optional) – a manual rescaling weight given to the loss of each batch element. If given, has to be a Tensor of size nbatch.

so you could need to calculate this tensor for each batch using the current targets (or use the pos_weight if applicable).


Distribution Error: The value argument must be within the support

Question:
So, I found this cool Normalizing flow tutorial in PyTorch and I was trying the first tut itself link here 10
import torch.distributions as distrib
import torch.distributions.transforms as transforms

x = np.linspace(-4, 4, 1000)
z = np.array(np.meshgrid(x, x)).transpose(1, 2, 0)
z = np.reshape(z, [z.shape[0] * z.shape[1], -1])

And the transformations like
# Initial distribution
q0 = distrib.MultivariateNormal(torch.zeros(2), torch.eye(2))
# Defining Affine Transformation
f1 = transforms.ExpTransform()
# Transforming
q1 = distrib.TransformedDistribution(q0, f1)

---> q1.log_prob(torch.Tensor(z))

The last line raises ValueError: The value argument must be within the support which is not the case in the notebook did something change? Because I tried running the whole notebook on colab and it broke there too and on my system too.
Any help what might be wrong?

Context: Any workaround for this?


Convert keras model architecture to Pytorch

Question:
I am trying to convert a keras model to pytorch for human activity recognition. The keras model could achieve up to 98% accuracy, while Pytorch model could achieve only ~60% accuracy. I couldn’t figure out the problem, first I was thinking of the padding=‘same’ of keras, but I have already adjusted the padding of pytorch already. Can you check what is wrong?
The keras code is as above
model = keras.Sequential()
model.add(layers.Input(shape=[100,12]))
model.add(layers.Conv1D(filters=32, kernel_size=3, padding="same"))
model.add(layers.BatchNormalization())
model.add(layers.ReLU())
model.add(layers.Conv1D(filters=64, kernel_size=3, padding="same"))
model.add(layers.BatchNormalization())
model.add(layers.ReLU())
model.add(layers.MaxPool1D(2))
model.add(layers.LSTM(64))
model.add(layers.Dense(units=128, activation='relu'))
model.add(layers.Dense(13, activation='softmax'))
model.summary()

and my pytorch model code is as below
class CNN(nn.Module):
    def __init__(self):
        super(CNN, self).__init__()
        seq_len = 100
        # output: [, 32, 100]
        self.conv1 = nn.Conv1d(seq_len, 32, kernel_size=3, stride=1, padding=1)
        self.bn1 = nn.BatchNorm1d(32)
        # output: [, 64, 100]
        self.conv2 = nn.Conv1d(32, 64, kernel_size=3, padding=1)
        self.bn2 = nn.BatchNorm1d(64)
        # output: [, 64, 50]
        self.mp = nn.MaxPool1d(kernel_size=2, stride=2)
        # output: [, 64]
        self.lstm = nn.LSTM(6, 64, 1)
        # output: [, 128]
        self.fc1 = nn.Linear(64, 128)
        # output: [, 13]
        self.fc2 = nn.Linear(128, 13)

    def forward(self, x):
        x = self.conv1(x)
        x = self.bn1(x)
        x = F.relu(x)

        x = self.conv2(x)
        x = self.bn2(x)
        x = F.relu(x)
        x = self.mp(x)

        out, _ = self.lstm(x)
        x = out[:, -1, :]

        x = self.fc1(x)
        x = F.relu(x)
        x = self.fc2(x)

        return x

Context:
I guess the issue could be created by the nn.LSTM layer in the default setup, which would expect an input in the shape [seq_len, batch_size, nb_features] and could thus process the inputs in a wrong way.
If you want to pass inputs with the batch dimension in dim0, you could create the modules with batch_first=True. The docs explain the input shapes a bit more.


Question:
We are currently implementing a WaveNet like structure to solve the problem of MNIST prediction, ie. given some part of the image, to reconstruct the rest. We’ve trained on the whole MNIST data set for starters, however we run into some problems regarding our model output estimate, namely that it is all gray.
From what we gather of info on PyTorch documentation we want to use binary cross entropy, and our model will use softmax on the output. Does Binary_cross_entropy use softmax, like Cross_entropy? We also tried to use the sigmoid, as the BCELoss function seemed to use it, but to no avail.

Context:
F.binary_cross_entropy is the functional API version of nn.BCELoss, so your model would need to use torch.sigmoid on its outputs. Also, I assume that you are working on a binary or multi-label classification.
If so, I would recommend to remove the sigmoid and use F.binary_cross_entropy_with_logits(input, target) instead.


Question:
Hello, I’m trying to use Posit<8,1> format in Pytorch with SoftPosit.
To start, I’m trying to load MNIST dataset and then, cast it to Posit8. My problem comes when I try to cast the X train to sp.posit8, apparently the program do it well, but after 2 minutes it gets blocked and after some time appears a message saying “kernel dead”.
This is the code I use:
Downloading the datasets in Pytorch format:
# define transforms
# transforms.ToTensor() automatically scales the images to [0,1] range
transforms = transforms.Compose([transforms.Resize((32, 32)),
                                 transforms.ToTensor()])

# download and create datasets
train_dataset = datasets.MNIST(root='mnist_data',
                               train=True,
                               transform=transforms,
                               download=True)

valid_dataset = datasets.MNIST(root='mnist_data',
                               train=False,
                               transform=transforms)

# define the data loaders
train_loader = DataLoader(dataset=train_dataset,
                          batch_size=BATCH_SIZE,
                          shuffle=True)

valid_loader = DataLoader(dataset=valid_dataset,
                          batch_size=BATCH_SIZE,
                          shuffle=False)

Here, I get the X and Y values of the train dataset and put them in numpy arrays:
X_train = []
Y_train = []
for x,y in train_loader:
    X_train.append(x.numpy())
    Y_train.append(y.numpy())
X_train = np.array(X_train, dtype=sp.posit8)
Y_train = np.array(Y_train, dtype=sp.posit8)

Finally, this is the part where the kernel dies:
aux = np.empty_like(X_train, dtype=sp.posit8)
for i in range(X_train.size):
    aux.flat[i] = sp.posit8(X_train.flat[i])

X_test = aux

Is there any solution to fix my kernel dead problem?
Thank you

Context:
I assume you are using a Jupyter notebook, which might not show the error message properly, but instead would kill the kernel and restart it.
Try to run the script in the terminal and check, if you would get a valid error message.


I need help ” TypeError: can’t convert CUDA tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first ”

Question:
i trying to run this plot but is said error: can’t convert CUDA tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first , anyone know why?
def plot_training_history(history):
  fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(18, 6))

  ax1.plot(history['train_loss'], label='train loss')
  ax1.plot(history['val_loss'], label='validation loss')

  ax1.xaxis.set_major_locator(MaxNLocator(integer=True))
  ax1.set_ylim([-0.05, 1.05])
  ax1.legend()
  ax1.set_ylabel('Loss')
  ax1.set_xlabel('Epoch')

  ax2.plot(history['train_acc'], label='train accuracy')
  ax2.plot(history['val_acc'], label='validation accuracy')

  ax2.xaxis.set_major_locator(MaxNLocator(integer=True))
  ax2.set_ylim([-0.05, 1.05])
  ax2.legend()

  ax2.set_ylabel('Accuracy')
  ax2.set_xlabel('Epoch')

  fig.suptitle('Training history')

Context:
Some aspects of matplotlib will try to convert the input you provide it into numpy. It can do it only if the provided input tensor is in cpu.
So, to begin with I would suggest you to try and do .cpu() to the inputs that you provide, that is, check if every value in the dict history that you are using is on cpu.


Question:
I have a dataset having 11k rows and 15 columns.
It’s a time series data, each row corresponds to one event in time, there could be multiple events in single day. Hence when creating data loader I did not use shuffle. I have a classification task with 7 classes. I used deep net for it and it didn’t give that good results and read that LSTM could be helpful as it has memory element. I wish to use LSTM in my model and hope that based on past data it captures the relationship and help me in predicting the classes better.
My question

what should be the seq length value in LSTM layer? Should it be 1? Corresponding to each row?
2)Also should I be initialising hidden state and cell state in each forward pass or use last computed hidden state and cell state value for next forward pass?
I read that weights are shared across timestep but since a batch of data is fed at a time how do these weights get shared?

Context:
number of events. “seq length” is never explicitly specified as RNNs work in “one step at a time” mode. you can even batch sequences of varying lengths with nn.utils.rnn.pack_padded_sequence. In general, data loader for RNNs is a bit tricky (as you may want to sample timeseries and/or subsequences, depending on task).
1b. you don’t pass states across iterations, as new mini-batch implies drawing a new set of samples (timeseries or random start points)
same as with other NNs, weights are not sample specific, but transform some data across batch dimension the same way. Sharing across timesteps additionally implies that you can make timesteps with new events for as long as you need.


About License for “c10.lib” and “libiompstubs5md.dll”

Question:
Dear Pytorch License contact,
I am writing this email to know the license for Pytorch.
It seems that Pytorch includes following unknown license files.
・torch/lib/c10.lib:
・torch/lib/libiompstubs5md.dll
nuget repository url:
NuGet Gallery | Opteamizer.Torch-Debug 2.0.0 3
NuGet Gallery | Opteamizer.Torch-Shared 2.0.0
Could you tell me the license for these files?
Sincerely,
Shohei Suenaga

Context: CC @jspisak, who might know the licenses of these files.


How to use PyTorch GPU acceleration-CNN- MNIST handwritten digit recognition

Question:
Hello everyone, I am learning pytroch recently and found this example from the Internet (PyTorch - CNN 卷積神經網絡 - MNIST手寫數字辨識 - HackMD 1).
I want to try GPU acceleration. I have crawled some information on the forum.
But I still can’t write it, so I would like to ask experienced people to tell me what changes need to be made to the code to achieve GPU computing.
Note: Because I want to visualize the accuracy and error percentage, when I read the article before and said that when I use plt, I need to change the data back to the cpu, so please help me pay attention to it, thank you.
Note:
Because I want to visualize the accuracy and error percentage,
when I read the article before and said that when I use plt,
I need to change the data back to the cpu, so please help me pay attention to it, thank you.
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.autograd import Variable
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt

from keras.datasets import mnist
(X_train, Y_train), (X_test, Y_test) = mnist.load_data()

X_train = X_train.astype('float32') / 255
X_test = X_test.astype('float32') / 255
Y_train
Y_test

features_train, features_test, targets_train, targets_test = train_test_split(X_train, Y_train, test_size = 0.2, random_state = 42)

featuresTrain = torch.from_numpy(features_train)
targetsTrain = torch.from_numpy(targets_train).type(torch.LongTensor) # data type is long

featuresTest = torch.from_numpy(features_test)
targetsTest = torch.from_numpy(targets_test).type(torch.LongTensor) # data type is long

# Pytorch train and test TensorDataset
train = torch.utils.data.TensorDataset(featuresTrain,targetsTrain)
test = torch.utils.data.TensorDataset(featuresTest,targetsTest)

# Hyper Parameters
# batch_size, epoch and iteration
LR = 0.01
batch_size = 100
n_iters = 10000
num_epochs = n_iters / (len(features_train) / batch_size)
num_epochs = int(num_epochs)

# Pytorch DataLoader
train_loader = torch.utils.data.DataLoader(train, batch_size = batch_size, shuffle = True)
test_loader = torch.utils.data.DataLoader(test, batch_size = batch_size, shuffle = True)

# Create CNN Model
class CNN_Model(nn.Module):
    def __init__(self):
        super(CNN_Model, self).__init__()
        # Convolution 1 , input_shape=(1,28,28)
        self.cnn1 = nn.Conv2d(in_channels=1, out_channels=16, kernel_size=5, stride=1, padding=0) #output_shape=(16,24,24)
        self.relu1 = nn.ReLU() # activation
        # Max pool 1
        self.maxpool1 = nn.MaxPool2d(kernel_size=2) #output_shape=(16,12,12)
        # Convolution 2
        self.cnn2 = nn.Conv2d(in_channels=16, out_channels=32, kernel_size=5, stride=1, padding=0) #output_shape=(32,8,8)
        self.relu2 = nn.ReLU() # activation
        # Max pool 2
        self.maxpool2 = nn.MaxPool2d(kernel_size=2) #output_shape=(32,4,4)
        # Fully connected 1 ,#input_shape=(32*4*4)
        self.fc1 = nn.Linear(32 * 4 * 4, 10)

    def forward(self, x):
        # Convolution 1
        out = self.cnn1(x)
        out = self.relu1(out)
        # Max pool 1
        out = self.maxpool1(out)
        # Convolution 2
        out = self.cnn2(out)
        out = self.relu2(out)
        # Max pool 2
        out = self.maxpool2(out)
        out = out.view(out.size(0), -1)
        # Linear function (readout)
        out = self.fc1(out)
        return out

model = CNN_Model()
print(model)
optimizer = torch.optim.Adam(model.parameters(), lr=LR)   # optimize all cnn parameters
loss_func = nn.CrossEntropyLoss()   # the target label is not one-hotted
input_shape = (-1,1,28,28)

def fit_model(model, loss_func, optimizer, input_shape, num_epochs, train_loader, test_loader):
    # Traning the Model
    #history-like list for store loss & acc value
    training_loss = []
    training_accuracy = []
    validation_loss = []
    validation_accuracy = []
    for epoch in range(num_epochs):
        #training model & store loss & acc / epoch
        correct_train = 0
        total_train = 0
        for i, (images, labels) in enumerate(train_loader):
            # 1.Define variables
            train = Variable(images.view(input_shape))
            labels = Variable(labels)
            # 2.Clear gradients
            optimizer.zero_grad()
            # 3.Forward propagation
            outputs = model(train)
            # 4.Calculate softmax and cross entropy loss
            train_loss = loss_func(outputs, labels)
            # 5.Calculate gradients
            train_loss.backward()
            # 6.Update parameters
            optimizer.step()
            # 7.Get predictions from the maximum value
            predicted = torch.max(outputs.data, 1)[1]
            # 8.Total number of labels
            total_train += len(labels)
            # 9.Total correct predictions
            correct_train += (predicted == labels).float().sum()
        #10.store val_acc / epoch
        train_accuracy = 100 * correct_train / float(total_train)
        training_accuracy.append(train_accuracy)
        # 11.store loss / epoch
        training_loss.append(train_loss.data)

        #evaluate model & store loss & acc / epoch
        correct_test = 0
        total_test = 0
        for images, labels in test_loader:
            # 1.Define variables
            test = Variable(images.view(input_shape))
            # 2.Forward propagation
            outputs = model(test)
            # 3.Calculate softmax and cross entropy loss
            val_loss = loss_func(outputs, labels)
            # 4.Get predictions from the maximum value
            predicted = torch.max(outputs.data, 1)[1]
            # 5.Total number of labels
            total_test += len(labels)
            # 6.Total correct predictions
            correct_test += (predicted == labels).float().sum()
        #6.store val_acc / epoch
        val_accuracy = 100 * correct_test / float(total_test)
        validation_accuracy.append(val_accuracy)
        # 11.store val_loss / epoch
        validation_loss.append(val_loss.data)
        print('Train Epoch: {}/{} Traing_Loss: {} Traing_acc: {:.6f}% Val_Loss: {} Val_accuracy: {:.6f}%'.format(epoch+1, num_epochs, train_loss.data, train_accuracy, val_loss.data, val_accuracy))
    return training_loss, training_accuracy, validation_loss, validation_accuracy

training_loss, training_accuracy, validation_loss, validation_accuracy = fit_model(model, loss_func, optimizer, input_shape, num_epochs, train_loader, test_loader)

# visualization
plt.plot(range(num_epochs), training_loss, 'b-', label='Training_loss')
plt.plot(range(num_epochs), validation_loss, 'g-', label='validation_loss')
plt.title('Training & Validation loss')
plt.xlabel('Number of epochs')
plt.ylabel('Loss')
plt.legend()
plt.show()
plt.plot(range(num_epochs), training_accuracy, 'b-', label='Training_accuracy')
plt.plot(range(num_epochs), validation_accuracy, 'g-', label='Validation_accuracy')
plt.title('Training & Validation accuracy')
plt.xlabel('Number of epochs')
plt.ylabel('Accuracy')
plt.legend()
plt.show()

Context:
Hi 浚瑋!

 111504:

what changes need to be made to the code to achieve GPU computing.

Pytorch tensors can be “moved” to the gpu so that computations
occur – greatly accelerated – on the gpu.
You can created a copy of a cpu tensor that resides on the gpu with:
my_gpu_tensor = my_cpu_tensor.cuda()

If you have a model that is derived from torch.nn.Module, you can
have it move its weights to the gpu with:
my_model = torch.nn.Linear (5, 10)   # Linear is a kind of Module
# my_model is initially on the cpu
my_model.cuda()   # move the model weights to the gpu
# my_model is now on the gpu

I need to change the data back to the cpu,

If you have a tensor on the gpu – for example the output of your
model that is running on the gpu – you can make a copy of it on
the cpu with:
my_cpu_tensor = my_gpu_tensor.cpu()

from torch.autograd import Variable

Your use of Variable suggests that the example you are working
with is quite old and is written for an old version of pytorch.
(Variable is an old wrapper for Tensor that keeps track of the
information needed for autograd.  Newer versions of pytorch – that
are better in lots of ways – have this functionality built directly into
Tensor so Variable is now deprecated.)
Make sure you are using a reasonably up-to-date version of pytorch
and base your learning on up-to-date example code.
The main pytorch web site has a number of (up-to-date) tutorials 2
that would be a good place to start.
Good luck.
K. Frank


Opacus: How to disable backward hook temporally for multiple backward pass

Question:
Hello,
I’m using Opacus for computing the per-sample gradient w.r.t the parameter. However, I also need to compute per-sample gradient of each logit w.r.t the input. Therefore I need to do back-propagation several times.  A minimal example is as follows
import torch
from opacus.grad_sample import GradSampleModule
from torch.autograd import grad

class Model(torch.nn.Module):
    def __init__(self, inputSize, outputSize):
        super(Model, self).__init__()
        self.linear = torch.nn.Linear(inputSize, outputSize)

    def forward(self, x):
        out = self.linear(x)
        return out

num_classes = 2
bs = 16

model = Model(10,num_classes)
op_model = GradSampleModule(model)

op_model.zero_grad()
X = torch.rand(bs,10)
X.requires_grad = True
grad_x = torch.zeros(bs,num_classes,10)

output = op_model(X) # bs * num_classes
for c in range(num_classes):
    grad_x[:,c,:] = grad(outputs=output[:,c], inputs=X,\
                         grad_outputs=torch.ones_like(output[:,c]),retain_graph=True)[0]

The error shows

IndexError: pop from empty list

It seems that Opacus won’t work if the number of backprop is greater than the number of forward pass, according to this post 3. However, in my use case (adversarial training) at some point I need to do backprop several times to compute input gradient (not parameter’s gradient). I wonder if it is possible to disable the grad_sample functionality temporarily, and enable it afterwards. I appreciate any suggestions on this.
Thanks!

Context:
Hi @Yuancheng_Xu!
The hook for computing grad_sample doesn’t work correctly with more than one backward pass.
I’m not sure I completely understand your use case, but I think you can do the following:
# initialize the vanilla model
model = Model(10,num_classes)
model.zero_grad()
X = torch.rand(bs,10)
X.requires_grad = True
grad_x = torch.zeros(bs,num_classes,10)

# compute the output and doutput/dinput
output = model(X) # bs * num_classes
for c in range(num_classes):
    grad_x[:,c,:] = grad(
        outputs=output[:,c],
        inputs=X,
        grad_outputs=torch.ones_like(output[:,c]),
        retain_graph=True
    )[0]

# add the hook & evaluate the model one more time to get the grad_sample
dp_model = GradSampleModule(model)
output = dp_model(X)

# ... apply gradient step with grad_sample

# remove hooks before the next iteration
# if you don't remove hooks, model(data) will be computed with grad_samples leading to the exception
# also you won't be able to recreate GradSampleModule(model) before removing hooks
dp_model.remove_hooks()

Note that using grads without properly added noise like in this case could break DP guarantees if you expected any.


ValueError: Unexpected grad_sample type: <class ‘NoneType’> when trying to train with differential privacy

Question:
Hello,
I am using PyTorch 1.10.0+cu102 and Opacus 1.0.0 and would like to use differential privacy for the training of an image classifier. When using optimizer.step() during training, I get the following ValueError: Unexpected grad_sample type: <class ‘NoneType’>
Specifically, this is my Traceback:
Traceback (most recent call last):
  File "run.py", line 102, in <module>
    sc.run_learning()
  File "/gpfs/home/user/diffp_project/run2.py", line 82, in run_learning
    le.run_learning_and_testing_process()
  File "/gpfs/home/user/diffp_project/LearningEnvironment.py", line 165, in run_learning_and_testing_process
    client.federated_round_local_train(model=self.global_model)
  File "/gpfs/home/user/diffp_project/Client.py", line 208, in federated_round_local_train
    epoch_loss_train = self.local_train(model=self.model)
  File "/gpfs/home/user/diffp_project/Client.py", line 176, in local_train
    running_loss = BatchIterator(model=model, phase=phase, Data_loader=train_loader, criterion=criterion, optimizer=optimizer, device=self.device)
  File "/gpfs/home/user/diffp_project/batchiterator.py", line 58, in BatchIterator
    optimizer.step()
  File "/home/user/miniconda3/envs/env5/lib/python3.7/site-packages/opacus/optimizers/optimizer.py", line 478, in step
    if self.pre_step():
  File "/home/user/miniconda3/envs/env5/lib/python3.7/site-packages/opacus/optimizers/optimizer.py", line 459, in pre_step
    self.clip_and_accumulate()
  File "/home/user/miniconda3/envs/env5/lib/python3.7/site-packages/opacus/optimizers/optimizer.py", line 366, in clip_and_accumulate
    g.view(len(g), -1).norm(2, dim=-1) for g in self.grad_samples
  File "/home/user/miniconda3/envs/env5/lib/python3.7/site-packages/opacus/optimizers/optimizer.py", line 313, in grad_samples
    ret.append(_get_flat_grad_sample(p))
  File "/home/user/miniconda3/envs/env5/lib/python3.7/site-packages/opacus/optimizers/optimizer.py", line 186, in _get_flat_grad_sample
    raise ValueError(f"Unexpected grad_sample type: {type(p.grad_sample)}")
ValueError: Unexpected grad_sample type: <class 'NoneType'>

Does anybody know how to fix it?
Here is some more information about my code:
The model is a torchvision.models.densenet121(pretrained=True), the optimizer is a torch.optim.SGD(self.model.parameters(), lr=0.1, momentum=0), the loss criterion is nn.BCELoss().to(self.device).
I also run model = ModuleValidator.fix(model), and of course model, optimizer, train_loader = self.privacy_engine.make_private(…) before the training.
Thanks in advance!

Context:
Hi @general,
Would you be able to share with us a reproducible code so we can debug it? I believe another user has reported this here:

      github.com/pytorch/opacus

      Getting "ValueError: Unexpected grad_sample type: <class 'NoneType'>" for self-supervised training. 4

        opened

        Dec 12, 2021

          vangorade

    Congratulations on the new opacus version release!

I am trying to train a con…trastive learning model on time-series data with opacus version 1.0 by following the official [tutorial ](https://github.com/pytorch/opacus/blob/main/tutorials/building_image_classifier.ipynb).

Error:
"ValueError: Unexpected grad_sample type: <class 'NoneType'>"

Full error:
    model_optimizer.step()  -> error starts from this line in my code.
  File "/mnt/f/AI_research/TS-TCC-main/opacus/opacus/optimizers/optimizer.py", line 478, in step
    if self.pre_step():
  File "/mnt/f/AI_research/TS-TCC-main/opacus/opacus/optimizers/optimizer.py", line 459, in pre_step
    self.clip_and_accumulate()
  File "/mnt/f/AI_research/TS-TCC-main/opacus/opacus/optimizers/optimizer.py", line 366, in clip_and_accumulate
    g.view(len(g), -1).norm(2, dim=-1) for g in self.grad_samples
  File "/mnt/f/AI_research/TS-TCC-main/opacus/opacus/optimizers/optimizer.py", line 313, in grad_samples
    ret.append(_get_flat_grad_sample(p))
  File "/mnt/f/AI_research/TS-TCC-main/opacus/opacus/optimizers/optimizer.py", line 186, in _get_flat_grad_sample
    raise ValueError(f"Unexpected grad_sample type: {type(p.grad_sample)}")
ValueError: Unexpected grad_sample type: <class 'NoneType'>

Thank you.


Opacus with large clipping norm performs differently from normal network

Question:
Hi, I compared two tests:

resnet20 on cifar10 with privacy-engine, the clipping norm is set to 10M. This should be equivalent to not doing clipping at all.

resnet20 on cifar10 without privacy-engine (noise-multiplier is set as 0), with exactly the same parameters as example 1.

Test 2 soon reached 92% accuracy while test 1 struggled to reach 85%. I then wrote another test where we created two models for the two above tests and made them train on the same data (with the same trainloader) simultaneously. Model 1 has optimizer 1 which is attached to a privacy-engine, while model 2 has optimizer 2 which is just a normal SGD optimizer.
The code looks something like this:
loss1.backward()
loss2.backward()
optimizer1.step()
optimizer2.step()
Before we call the step() functions, the param.grad are exactly the same between the two models. However, after we called the step() functions, there are approximately a 3% difference between the param.grad of the two models.
Is this because pytorch’s default way of computing gradient is different from opacus even when the clipping value is 10 million? Or is it because of accuracy loss during opacus computations?

Context: Are you sure you use the same exact network in both cases? The “canonical” Resnet20 includes batch normalization, which is incompatible with DP-SGD.


Question:
I’m trying to utilise opacus with the PyTorch Lightning framework which we use as a wrapper around a lot of our models. I can see that there was an effort to integrate this partially into PyTorch Lightning late last year 5 but this seems to have stalled due to lack of bandwidth.
I’ve created a simple MVP 9 but there seems to be a compatibility problem with even this simple model; it throws AttributeError: 'Parameter' object has no attribute 'grad_sample' as soon as it hits the optimization step.
What’s the likely underlying cause of this? I can see on the opacus GitHub that similar errors have been encountered before where it’s been caused by unsupported layers but as the gist shows, this model is incredibly simple so I don’t think it’s any of the layers.
This is with:
opacus==0.11.0
pytorch-lightning==1.2.1
torch==1.7.1
torchaudio==0.7.2
torchvision==0.8.2

Context: Hi James! Would you have time to file a bug and share a colab so I can take a look? Integrating with Lightning is indeed on our plate


Unexpected grad_sample type: <class ‘NoneType’>

Question:
I was trying to use opacus with DCGAN,while i am training Discriminator with fake images,
In the line of “opt_D.step()”,such value error happened,i can’t figure out what’s going wrong?PLEASE HELP

162604723×297 21.8 KB

Context: 162543730×340 16.7 KB


CUDA memory builds up linearly with batches on privacy engine

Question:
I am training on MNIST images using resnet-18 architecture on a 16 GB GPU memory machine. Everything works perfectly if I do not attach the privacy engine to my optimizer, each batch takes about 1-2 GB GPU memory, which is flushed out after processing each batch, and hence the total GPU memory consumption stays around 2 GB throughout the process of training.
However, when I attach the privacy engine to my optimizer, the used GPU memory keeps accumulating with each batch. Hence, with the processing of each batch, the used GPU memory increases, and soon I receive CUDA out of memory error.
I am not able to understand why GPU memory is not being freed at the end of every batch upon the inclusion of the privacy engine.

Context: Do you observe memory leak during training or test? Also, are you running the latest version of Opacus?


Question:
Hi!
I’m using Opacus to train my model. The version before 1.0.0 works fine without dataloader. However, it seems that the new version privacy engine requires one at initialization. Is there any way to avoid this?
Thx!

Context:
Hi!
Thanks for using opacus.
There’s certainly a way to use opacus without DataLoader - please refer to our Migration Guide, section “No DataLoader” 2
Feel free to ask any further questions here


How to store the state and resume the state of the PrivacyEngine?

Question:
Hi,
I run my computations on a server cluster where computation jobs have a time limit, but my learning process of multiple epochs typically takes longer than this time limit. Therefore, I regularly store the state of my computations (i.e., after every epoch), and the resume the computations when the job finished and I started a new one.
This especially includes using the .load_state_dict() function of the model and the optimizer.
Now I would like to integrate differential privacy with Opacus into my computations, and I am asking myself whether and how I can store the states of Opacus, and resume it? Specifically, my questions are:

Can I just torch.save() the PrivacyEngine object?
If the answer to 1) is no, is there a .state_dict property and .load_state_dict() function for the PrivacyEngine?
Can I alternatively just run the model.load_state_dict() and optimizer.load_state_dict(), then update these by running privacy_engine.make_private(), and use the objects as normal?
In the case of 3), can I store the epsilon = privacy_engine.get_epsilon(DELTA) after every epoch (along with the model state and the optimizer state), and add the previous epsilon value to the epsilon value obtained after resuming the calculations? Or is this not an additive relation and I would end up with a wrong epsilon value when resuming the calculations?

Thank you very much in advance!

Context:
Hi,
Thanks for your question - ability to save/load checkpoints is an important feature, and it’s good for us to have some input on how people could be using it.
While we’re considering how to include this into Opacus API here’s what you need to know to make it work today.

PrivacyEngine doesn’t maintain links to model, optimizer, or data_loader. The only important state maintained by PrivacyEngine is accountant. Accountant’s state is just a list of numerical tuples, so torch.save() or any other pickle mechanism should do. That said, you probably should save the accountant (privacy_engine.accountant), not the privacy engine itself. That’s because we also maintain the link to the dataset used in the first call to the make_private() method - to do a sanity check the dataset is not being swapped in the middle (accounting is performed on per-dataset basis)
While GradSampleModule functionally is just a wrapper around nn.Module, saving/loading probably doesn’t work for them out of the box. I’d say your best bet is to save/load underlying model and then wrap it with GradSampleModule every time you’re restoring from the checkpoint.

To summarise, here are the steps you need to take
On saving:

Save accountant: torch.save(privacy_engine.accountant)

Save model: torch.save(model. _module.state_dict())

If your optimizer has state (e.g. dynamic lr or momentum) - save wrapped optimizer: optimizer.original_optimizer.state_dict()

On loading:

Initialize empty PrivacyEngine
Load accountant and replace brand new with the one you’ve just initialized: privacy_engine.accountant = accountant_you_have_just_loaded_from_checkpoint

Load your non-private nn.Module as normal
Load your non-private optimizer as normal
Pass loaded model and optimzier to privacy_engine.make_private()


Get per-sample gradient without clipping and noise adding

Question:
Hi,
Given a training process I need, at each step, to get all gradients tensors associated to each individual sample of the batch; then I need to perform some operation on each of this gradients, finally collect them together and perform the .optimizer.step().
I found that Opacus  1  could fit my problem, but:

I don’t need nor the gradient clipping neither the noise adding

on the other hand I need to access the set of gradients associated to each batch’s element and perform some operation on them; after these operations I’ll get a single gradient that will be used for the weight update (.optimizer.step()).
Is it possible to do this with Opacus library?

Context:
Hi @Torcione
Yes, you can use opacus for that; Take a look at GradSampleModule (opacus/grad_sample/grad_sample_module.py)
It’s a wrapper around nn.Module that encapsulates per sample gradient computation. When you wrap your model with GradSampleModule, each trainable parameter will get .grad_sample attribute containing per-sample gradients. No noise or clipping is performed.
Hope this helps


Question:
Hey everyone, I attempt to accumulate gradients in training to save GPU memory.
The training loop works quite well without privacy_engine
opt.zero_grad()
for i, (input, target) in enumerate(dataset):
    pred = net(input)
    loss = crit(pred, target)
    # one graph is created here
    loss.backward()
    # graph is cleared here
    if (i+1)%10 == 0:
        # every 10 iterations of batches of size 10
        opt.step()
        opt.zero_grad()

However, when I attach the  privacy_engine to the optimizer, CUDA is out of memory.
Do anyone know how to solve this problem? Thanks in advance.

Context:
Hi!
It is expected that Opacus has a certain memory overhear. At the very least, we have to store per-sample gradients for all model parameters - that alone increases the memory required to store gradient by the factor of batch size.
In order to address that I suggest using virtual_step() method in the optimizer.
It does gradient clipping and accumulation (thus saving memory), but doesn’t do the actual optimizer step.
Your code would look smth like this:
opt.zero_grad()
for i, (input, target) in enumerate(dataset):
    pred = net(input)
    loss = crit(pred, target)
    # one graph is created here
    loss.backward()
    # graph is cleared here
    if (i+1)%10 == 0:
        # every 10 iterations of batches of size 10
        opt.step()
        opt.zero_grad()
    else:
        opt.virtual_step()

For more examples using virtual_step see our Cifar10 tutorial: opacus/cifar10.py at main · pytorch/opacus · GitHub 1


Question:
Hey there, we are trying to train a private version of a particular model which uses nn.Parameter and getting the error torchdp.dp_model_inspector.IncompatibleModuleException.
In particular, the parameters are defined for the model here 10 and used in the forward function here 4. To the best of our knowledge these parameters and associated operations preserve privacy because they don’t compute any aggregate batch statistics. What would be the recommended way about being able to train with this model definition.
Is there some sort of workaround we could do to wrap these lines in a valid module? Do we need to wait for the team to add an accepted module to opacus.SUPPORTED_LAYERS?

Context:
Hi Chris!
Luckily you don’t need to fork nor wait for the team to change that for you. You will however have the responsibility of writing a function to compute grad_sample for your layer. Once you write it, you simply register it for your layer using @register_grad_sampler 7 and you should be good to go!


Question:
Hi all,
I have followed tutorials regards DP Image Classification using Resnet18.
I have some questions:

When a model has many layers, it wasn’t able to convergence under DP. Are there any recommended approaches to overcome this problem for large models with many Fully connected layers?
When I decreased batch_size using the same model (due to memory size 8GB), the loss goes up to 80-200. That’s mean it is challenging to make a large batch with DP. Large batch size sometimes helps to improve model accuracy.
With trying different DL models, training non-private model takes a reasonable time (e.g., 6 minutes) compared to the private model (19 minutes) with lower accuracy.

Is DP-SGD very slow due to per-sample gradients? Is there any way to speed up processing time?
Can we achieve a comparable accuracy with the baseline model under a modest privacy budget?
Is the DP-Deep Learning model more sensitive to hyperparameters like batch size and noise level or the structure of NN?
Thanks,

Context:
Hello @NBu,
I really don’t know how this post went unattended. Sorry for the delay.

When a model has many layers, it wasn’t able to convergence under DP. Are there any recommended approaches to overcome this problem for large models with many Fully connected layers?

Do you mind sharing an example notebook of this? Hyper-parameter tuning should typically help with these cases. Some tips: FAQ · Opacus

When I decreased batch_size using the same model (due to memory size 8GB), the loss goes up to 80-200. That’s mean it is challenging to make a large batch with DP. Large batch size sometimes helps to improve model accuracy.

The memory requirement is an unfortunate consequence of maintaining per-sample gradients. Opacus provides a concept of virtual_batch to overcome this issue. Please try it out if you haven’t already.  FAQ · Opacus

Is DP-SGD very slow due to per-sample gradients? Is there any way to speed up processing time?

There is some overhead to computing per-sample gradients. The speed degradation depends on the layers as well (eg Linear vs LSTM). We can take a look at your notebook to make some suggestions.

Can we achieve a comparable accuracy with the baseline model under a modest privacy budget?

Depends a lot on the task. As you can see on https://opacus.ai 1, the answer is a resounding yes for MNIST with a small network. However, there is a ~20pp gap in accuracy for CIFAR-10 with ResNet18.
Bridging this gap is an active area of research, and we whole heartedly welcome your ideas and contributions. One of the main goals of open-sourcing opacus is to help push this research.


Compute per sample gradient for a normal model

Question: Hello, How to compute a per sample gradient for a usual model using Opacus? In this case, we don’t care about the privacy issue.

Context:
Hello,
Excellent question. We recently wrote a blog about this: Differential Privacy Series Part 2 | Efficient Per-Sample Gradient Computation in Opacus | by PyTorch | PyTorch | Sep, 2021 | Medium 14


Deprecation warning with latest torch and opacus

Question:
With the latest versions:
opacus==0.14.0
torch==1.9.0

I’m getting the following deprecation warnings when running the CIFAR10 example:
/local_disk0/.ephemeral_nfs/envs/pythonEnv-4ed4226e-cb69-4f21-bbff-ba9e9ab8fc55/lib/python3.7/site-packages/torch/nn/modules/module.py:974: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
  warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "

Is this the expected behavior? Should I be concerned with the warning saying This hook will be missing some grad_input?
Thank you!

Context:
Hi!
Yeah, that’s somewhat expected - we never migrated to register_full_backward_hook  since it was released in PyTorch 1.8.
We’ll fix that soon (thanks for raising this!). But in the meantime, as  far as I’m aware, this shouldn’t create any problems (at least with 1.9) - all our tests are passing and we never received any reports of this causing any issues


IndexError: pop from empty list in grad_sample_module.py for opacus version > 0.9

Question:
Hi,
Im using Opacus to make CT-GAN (GitHub - sdv-dev/CTGAN: Conditional GAN for generating synthetic tabular data.) differntial private.
There is already an implementation who does this:  (smartnoise-sdk/dpctgan.py at main · opendp/smartnoise-sdk · GitHub 6)
However, they use an older version of opacus (v0.9) and CTGAN(v.0.2.2.dev1).
I used their method to make the newest version of CTGAN differential private with the newest opacus version. Unfortunatly i run into the following error:
.../CTGAN_DP/DP_CTGAN.py", line 309, in fit
    loss_d.backward()
  File ".../lib/python3.7/site-packages/torch/tensor.py", line 221, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File ".../lib/python3.7/site-packages/torch/autograd/__init__.py", line 132, in backward
    allow_unreachable=True)  # allow_unreachable flag
  File ".../python3.7/site-packages/opacus/grad_sample/grad_sample_module.py", line 197, in capture_backprops_hook
    module, backprops, loss_reduction, batch_first
  File ".../python3.7/site-packages/opacus/grad_sample/grad_sample_module.py", line 234, in rearrange_grad_samples
    A = module.activations.pop()
IndexError: pop from empty list

The function in question is:
 def rearrange_grad_samples(
        self,
        module: nn.Module,
        backprops: torch.Tensor,
        loss_reduction: str,
        batch_first: bool,
    ) -> Tuple[torch.Tensor, torch.Tensor]:
        """
        Rearrange activations and grad_samples based on loss reduction and batch dim

        Args:
            module: the module for which per-sample gradients are computed
            backprops: the captured backprops
            loss_reduction: either "mean" or "sum" depending on whether backpropped
            loss was averaged or summed over batch
            batch_first: True is batch dimension is first
        """
        if not hasattr(module, "activations"):
            raise ValueError(
                f"No activations detected for {type(module)},"
                " run forward after add_hooks(model)"
            )

        batch_dim = 0 if batch_first or type(module) is LSTMLinear else 1

        if isinstance(module.activations, list):
                A = module.activations.pop()
        else:
            A = module.activations

        if not hasattr(module, "max_batch_len"):
            # For packed sequences, max_batch_len is set in the forward of the model (e.g. the LSTM)
            # Otherwise we infer it here
            module.max_batch_len = _get_batch_size(module, A, batch_dim)

        n = module.max_batch_len
        if loss_reduction == "mean":
            B = backprops * n
        elif loss_reduction == "sum":
            B = backprops
        else:
            raise ValueError(
                f"loss_reduction = {loss_reduction}. Only 'sum' and 'mean' losses are supported"
            )

        # No matter where the batch dimension was, .grad_samples will *always* put it in the first dim
        if batch_dim != 0:
            A = A.permute([batch_dim] + [x for x in range(A.dim()) if x != batch_dim])
            B = B.permute([batch_dim] + [x for x in range(B.dim()) if x != batch_dim])

        return A, B

This does not happen with opacus v.09.
I investigated and found that module.activations is popped until only an empty list for the first module  is left and then produces this error.
I hacked my way around this issue as follows:
  if isinstance(module.activations, list):
            #print(len(module.activations))
            if len(module.activations) > 1:
                A = module.activations.pop()
            else:
                A = module.activations[0]
        else:
            A = module.activations

Meaning, if  module.activations is left with one element instead of an empty list at least training works.
My question is: Am I breaking anything important doing this and could this be a potential subcase which was not accounted for.
Or do I have to change something else in the model ?
The model I try to train with opacus is basically just a composition of  n*(nn.Liner nn.Relu nn.Dropout) which should be fine I think.
Thanks for a reply
Have a great day.

Context:
@shaanchandra From the discussion above, it seems that there are two potential solutions:

As @knilox mentioned above, remove the gradient regularization loss entirely - this may seem that we are not faithfully reproducing the original CTGAN, however DP eventually clips the gradients to bound SGD sensitivity, automatically introducing some regularization. Further, it appears that the smartnoise-sdk implementation which is being followed here already does this.
Keep the gradient regularization loss but do not compute per-sample gradients during the call to autograd.grad (basically keep the backward hook disabled during that time) as it is for estimation of gradients and not for updating the network. We’ve tested this out and it works initially, however it seems to be creating a new issue during the .step() where it complains that the norms are of unequal length.
So our recommendation is do (1). Option (2)'s error is probably unrelated and is also a fix.


Question:
for p, clip_value in zip(params, clip_values):
    noise = self._generate_noise(clip_value, p)
    if self.loss_reduction == "mean":
        noise /= batch_size
    if self.rank == 0:
        p.grad += noise

the noise is added to the averaged gradient, why should it be divided by the batch size?
since the output is the final grad, should it be added with noise (without being divided) directly?

Context:
Compare two cases. If there’s no reduction (self.loss_reduction == “sum”), then we want to add noise calibrated to the clipping norm X. Indeed, let the gradients be g_1,…,g_B. The sum is g_1 + … + g_B, and to make it private the additive noise is sampled from the Gaussian distribution N(0, sigma^2 * C^2) so that it masks presence or absence of any one gradient vector.
If the reduction function is mean, then the output is (g_1 + … + g_B) / B. What should the additive noise be in this case? I think it’s pretty obvious that it is must be a scaled down (by a factor of B) noise from before. The only difference between the two cases is the scaling parameter, and it should be applied equally both to the sensitive inputs and the noise. Right?


Question:
Hi All,
For an imbalanced dataset, can we first use oversampling to balance the two classes, and then use Opacus for DP training over the artificially balanced dataset?

Context:
Hi @PeterCheng and thanks for your question.
You can certainly do that, but it would affect how you would interpret privacy for the trained model.
Opacus provides privacy guarantees with respect to each individual sample in the dataset. We also use privacy amplification by subsampling - i.e. the epsilon guarantees directly tied to the probability of being sampled for a minibatch.
When you oversample, you essentially increase sampling rate for that particular data record.
To get an eps estimate for oversampled instances, you need to modify PrivacyEngine.get_privacy_spent method and use adjusted sample rate


Question:
Hello,
Is there any example for DP regression in Opacus please?

Context:
Hi @AliSh, thank you for your questions. I’m presuming you are looking for an example of linear regression task. While we do not have an example in the Opacus repo at the moment, it should still work out of the box just and the workflow looks the same as that of other examples. (DP-SGD clips and noises gradients rather than the outputs).
Please let us know if you are running into any issues there.
P.S.: We do welcome and appreciate pull-request if you’d like to contribute an example


No gradient in layers text classification tutorial

Question:
Hi!
I was running the tutorial on text classification, exactly as in opacus/building_text_classifier.ipynb at master · pytorch/opacus · GitHub 6, but I get the following error when I try to train:
AttributeError: The following layers do not have gradients: [‘module.bert.encoder.layer.11.attention.self.query.weight’, ‘module.bert.encoder.layer.11.attention.self.query.bias’, ‘module.bert.encoder.layer.11.attention.self.key.weight’, ‘module.bert.encoder.layer.11.attention.self.key.bias’, ‘module.bert.encoder.layer.11.attention.self.value.weight’, ‘module.bert.encoder.layer.11.attention.self.value.bias’, ‘module.bert.encoder.layer.11.attention.output.dense.weight’, ‘module.bert.encoder.layer.11.attention.output.dense.bias’, ‘module.bert.encoder.layer.11.attention.output.LayerNorm.weight’, ‘module.bert.encoder.layer.11.attention.output.LayerNorm.bias’, ‘module.bert.encoder.layer.11.intermediate.dense.weight’, ‘module.bert.encoder.layer.11.intermediate.dense.bias’, ‘module.bert.encoder.layer.11.output.dense.weight’, ‘module.bert.encoder.layer.11.output.dense.bias’, ‘module.bert.encoder.layer.11.output.LayerNorm.weight’, ‘module.bert.encoder.layer.11.output.LayerNorm.bias’, ‘module.bert.pooler.dense.weight’, ‘module.bert.pooler.dense.bias’, ‘module.classifier.weight’, ‘module.classifier.bias’]. Are you sure they were included in the backward pass?
Could someone help me understand why this is happening?
I’m on ubuntu and am using python 3.8.5
cheers!

Context:
Based on cell 8 it seems you are freezing some layers and train only others:
trainable_layers = [model.bert.encoder.layer[-1], model.bert.pooler, model.classifier]
total_params = 0
trainable_params = 0

for p in model.parameters():
        p.requires_grad = False
        total_params += p.numel()

for layer in trainable_layers:
    for p in layer.parameters():
        p.requires_grad = True
        trainable_params += p.numel()

print(f"Total parameters count: {total_params}") # ~108M
print(f"Trainable parameters count: {trainable_params}") # ~7M

so I would assume that the frozen parameters do not have valid gradients.
I’m however unsure where this message is raised from and if it’s an error etc. so could you explain the issue a bit more?


Gradient per example before and after clipping

Question:
Hello,
I wonder if there is an option in Opacus to access the per example gradient before and after clipping during the training please?
Thanks,
Ali

Context:
Hi
Accessing per sample gradients before clipping is easy - they’re available between loss.backward() and optimizer.step() calls. Backward pass calculates per sample gradients and stores them in parameter.grad_sample attribute. Optimizer step then does the clipping and aggregation, and cleans up the gradients.
For example:
m = Module()
optimizer = optim.SGD(m.parameters(), <...>)
privacy_engine = PrivacyEngine(<...>)
privacy_engine.attach(optimizer)

<...>
output = m(data)
loss = criterion(data, labels)
loss.backward()
print(m.fc.weight.grad_sample) # print per sample gradients
optimizer.step()

Post-clip values are more tricky - it’s not something we support out of the box.
optimizer.step() does three things at once:

clips per sample gradients
accumulates per sample gradients into parameter.grad

adds noise

Which means that there’s no easy way to access intermediate state after clipping, but before accumulation and noising.
I suppose, the easiest way to get post-clip values would be to take pre-clip values and do the clipping yourself, outside of opacus code.
All you need to do is replicate a small bit from opacus/opacus/per_sample_gradient_clip.py:clip_and_accumulate():
# step 0 : calculate the layer norms
all_norms = calc_sample_norms(
  named_params=self._named_grad_samples(),
)

# step 1: calculate the clipping factors based on the noise
clipping_factor = self.norm_clipper.calc_clipping_factors(all_norms)

You can then simply multiply your pre-clipping per-sample gradients p.grad_sample by clipping_factor tensor to get the same clipping that’s happening inside opacus.
Hope this helps


Question:
Hi, I am enjoying using the opacus package to apply differential privacy to the training process of my models, I am struggling to get it to work with my TVAE implementation though, could someone let me know why I get an Incompatible Module Exception, I am using similar modules to in all my other generative models. See my code below:
import numpy as np
import torch
from torch.nn import Linear, Module, Parameter, ReLU, Sequential
from torch.nn.functional import cross_entropy
from torch.optim import Adam
from torch.utils.data import DataLoader, TensorDataset

from models.CTGAN import DataTransformer, BaseSynthesiser
from .utils import GeneralTransformer, DPSynthesiser

import opacus
from opacus import autograd_grad_sample, PrivacyEngine, utils
import dill

class Encoder(Module):
    def __init__(self, data_dim, compress_dims, embedding_dim):
        super(Encoder, self).__init__()
        dim = data_dim
        seq = []
        for item in list(compress_dims):
            seq += [Linear(dim, item), ReLU()]
            dim = item
        self.seq = Sequential(*seq)
        self.fc1 = Linear(dim, embedding_dim)
        self.fc2 = Linear(dim, embedding_dim)

    def forward(self, input):
        feature = self.seq(input)
        mu = self.fc1(feature)
        logvar = self.fc2(feature)
        std = torch.exp(0.5 * logvar)
        return mu, std, logvar

class Decoder(Module):
    def __init__(self, embedding_dim, decompress_dims, data_dim):
        super(Decoder, self).__init__()
        dim = embedding_dim
        seq = []
        for item in list(decompress_dims):
            seq += [Linear(dim, item), ReLU()]
            dim = item

        seq.append(Linear(dim, data_dim))
        self.seq = Sequential(*seq)
        self.sigma = Parameter(torch.ones(data_dim) * 0.1)

    def forward(self, input):
        return self.seq(input), self.sigma

def loss_function(recon_x, x, sigmas, mu, logvar, output_info, factor):
    st = 0
    loss = []
    for column_info in output_info:
        for span_info in column_info:
            if len(column_info) != 1 or span_info.activation_fn != "softmax":
                ed = st + span_info.dim
                std = sigmas[st]
                loss.append(((x[:, st] - torch.tanh(recon_x[:, st])) ** 2 / 2 / (std ** 2)).sum())
                loss.append(torch.log(std) * x.size()[0])
                st = ed

            else:
                ed = st + span_info.dim
                loss.append(cross_entropy(recon_x[:, st:ed], torch.argmax(x[:, st:ed], dim=-1), reduction="sum"))
                st = ed

    assert st == recon_x.size()[1]
    KLD = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())
    return sum(loss) * factor / x.size()[0], KLD / x.size()[0]

class DPTVAE(BaseSynthesiser):

    def __init__(
        self,
        embedding_dim=128,
        compress_dims=(128, 128),
        decompress_dims=(128, 128),
        l2scale=1e-5,
        batch_size=500,
        disabled_dp=False,
        delta=1e-5,
        noise_multiplier=3.5,
        max_per_sample_grad_norm=1.0,
        epsilon=1.0,
        iterations=300,
        verbose=True,
    ):

        self.embedding_dim = embedding_dim
        self.compress_dims = compress_dims
        self.decompress_dims = decompress_dims

        self.l2scale = l2scale
        self.batch_size = batch_size
        self.loss_factor = 2
        self.iterations = iterations
        self.device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

        # opacus parameters
        self.noise_multiplier = noise_multiplier
        self.disabled_dp = disabled_dp
        self.delta = delta
        self.max_per_sample_grad_norm = max_per_sample_grad_norm
        self.epsilon = epsilon
        self.epsilon_list = []
        self.alpha_list = []
        self.loss_list = []
        self.verbose = verbose

    def train(self, data, categorical_columns=None, ordinal_columns=None, update_epsilon=None, verbose=False):
        self.transformer = DataTransformer()
        self.transformer.fit(data, discrete_columns=categorical_columns)
        data = self.transformer.transform(data)
        dataset = TensorDataset(torch.from_numpy(data.astype("float32")).to(self.device))
        loader = DataLoader(dataset, batch_size=self.batch_size, shuffle=True, drop_last=True)

        data_dim = self.transformer.output_dimensions
        self.encoder = Encoder(data_dim, self.compress_dims, self.embedding_dim).to(self.device)
        self.decoder = Decoder(self.embedding_dim, self.compress_dims, data_dim).to(self.device)
        self.optimizerAE = Adam(list(self.encoder.parameters()) + list(self.decoder.parameters()), weight_decay=self.l2scale)

        privacy_engine = opacus.PrivacyEngine(
            self.decoder,
            batch_size=self.batch_size,
            sample_size=data.shape[0],
            alphas=[1 + x / 10.0 for x in range(1, 100)] + list(range(12, 64)),
            target_delta=self.delta,
            noise_multiplier=self.noise_multiplier,
            max_grad_norm=self.max_per_sample_grad_norm,
            clip_per_layer=True,
        )

        if not self.disabled_dp:
            privacy_engine.attach(self.optimizerAE)

        if hasattr(self, "privacy_engine"):
            epsilon, best_alpha = self.optimizerAE.privacy_engine.get_privacy_spent(self.delta)
        else:
            epsilon = 0

        for i in range(self.iterations):

            if not self.disabled_dp:
                if self.epsilon < epsilon:
                    break

            for id_, data_ in enumerate(loader):
                self.optimizerAE.zero_grad()
                real = data_[0].to(self.device)
                mu, std, logvar = self.encoder(real)
                eps = torch.randn_like(std)
                emb = eps * std + mu
                rec, sigmas = self.decoder(emb)
                loss_1, loss_2 = loss_function(rec, real, sigmas, mu, logvar, self.transformer.output_info_list, self.loss_factor)
                loss = loss_1 + loss_2
                loss.backward()
                self.optimizerAE.step()
                self.decoder.sigma.data_.clamp_(0.01, 1.0)

                if not self.disabled_dp:
                    for p in self.decoder.parameters():
                        if hasattr(p, "grad_sample"):
                            del p.grad_sample

                    epsilon, best_alpha = self.optimizerAE.privacy_engine.get_privacy_spent(self.delta)

                    self.epsilon_list.append(epsilon)
                    self.alpha_list.append(best_alpha)

            if verbose:
                print("eps: {:f} \t alpha: {:f} \t Loss: {:f}".format(epsilon, best_alpha, loss.detach().cpu()))

            if not self.disabled_dp:
                if self.epsilon < epsilon:
                    break

            self.loss_list.append(loss)

        privacy_engine.detach()
        self.privacy_engine = privacy_engine
        self.state_dict = self.optimizerAE.state_dict()

        return self.loss_list, self.epsilon_list, self.alpha_list

    def sample(self, samples):
        self.decoder.eval()

        steps = samples // self.batch_size + 1
        data_ = []
        for _ in range(steps):
            mean = torch.zeros(self.batch_size, self.embedding_dim)
            std = mean + 1
            noise = torch.normal(mean=mean, std=std).to(self.device)
            fake, sigmas = self.decoder(noise)
            fake = torch.tanh(fake)
            data_.append(fake.detach().cpu().numpy())

        data_ = np.concatenate(data_, axis=0)
        data_ = data_[:samples]
        return self.transformer.inverse_transform(data_, sigmas.detach().cpu().numpy())

    def set_device(self, device):
        self.device = device
        self.decoder.to(self.device)

    def save(self, path):
        assert hasattr(self, "data_sampler")

        # always save a cpu model.
        device_bak = self.device
        self.device = torch.device("cpu")
        self.encoder.to(self.device)
        self.decoder.to(self.device)

        torch.save(self, path, pickle_module=dill)

        self.device = device_bak
        self.encoder.to(self.device)
        self.decoder.to(self.device)

    @classmethod
    def load(cls, path):
        model = torch.load(path)
        model.device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
        model.encoder.to(model.device)
        model.decoder.to(model.device)
        return model

def DPTVAE_runner(seed, run_name, epsilon, delta, n_s, df, batch_size, save, load, categorical_columns, ordinal_columns, iterations, noise_multiplier):

    if not load:
        gan = DPSynthesiser(
            DPTVAE(
                batch_size=batch_size,
                iterations=iterations,
                delta=delta,
                noise_multiplier=noise_multiplier,
            ),
            GeneralTransformer(),
            epsilon=epsilon,
        )
        gan.fit(df, categorical_columns=categorical_columns, ordinal_columns=ordinal_columns, verbose=True, seed=seed)
    else:
        gan = DPTVAE.load(load)

    df_synth = gan.sample(n_s)

    if save:
        gan.save(save)

    return df_synth, None

Any help would be appreciated, thanks!

Context:
Could you post the error message you are receiving?
I’m not familiar with Opacus so I don’t know, if you are facing a PyTorch error or if it’s related to Opacus.


Question:
Hello!
I have a question about Gradient Clipping, that arises from the following principles of privacy accounting and DP-SGD:
The RDP calculation for each step in training is based on the ratio between maximum norm bound of the gradients and the std. deviation of the noise being added to them. This ratio is known as the noise multiplier. As long as the ratio stays the same, the privacy guarantee for a given real valued function does not change. So if I want to increase the maximum norm bound (sensitivity) of a real valued function, the noise std. dev. just has to be scaled by the same amount to satisfy the same privacy. #11 (see also Opacus Issue #11 , as well as Proposition 7 / Corollary 3 in https://arxiv.org/pdf/1702.07476.pdf 3)
Given this, I want to discuss the following example:
Suppose (for the sake of simplicity) that I have chosen a norm bound of B = 1, and that the corresponding noise std. dev. sigma is also 1. The noise multiplier z = sigma/B = 1, and this real valued function then satisfies (alpha, alpha / 2*z^2)-RDP.
Consider then the following two cases during training:

If the gradient is of size 1 at a particular step in training, the noise fits the exact sensitivity of the gradient and the privacy is accounted for in a reasonable way.
However, If the gradient is less than the norm bound, let’s say of size 0.5, the noise of scale 1 suddenly is too big for the now smaller sensitivity. As stated above, B and sigma could be scaled down to 0.5 as well to satisfy the same privacy guarantee as before. Worded differently, if for this step we would change B = 0.5 (which is just as valid a clipping bound as 1 and yields the same update to the gradient) but keep sigma = 1, this would satisfy a different privacy guarantee while providing the same update to the parameters (as having B=1, sigma=1). More specifically the guarantee should be equivalent to doubling the size of z resulting in alpha, alpha / 2*(2*z)^2) = (alpha, alpha / 8*z^2)-RDP.

My question now is, is there an obvious reason as to why this is not considered in privacy accounting? It does not seem to me that the accountant takes into consideration the actual scale of gradients or scales noise accordingly. The clipping threshold and noise multiplier are constant hyperparameters that are to be freely chosen by the user of Opacus. Because these are constant the noise that is added to the gradients is also always constant. As the sizes of the gradients most definitely are not, this leads me to believe that the privacy calculation sometimes would yield the first case for which the noise is correctly scaled and at other times the second case listed above for which the noise is not accurate (or rather always pessimistic) and that we add too much noise for a given guarantee, hurting the models utility.
Could you address this concern and whether it is possible to mitigate this using something like an adaptive clipping bound/noise during training? Or is there something I am missing?
Thanks in advance!

Context:
Hi!
You are right: indeed the accountant does not consider the scale of gradients of each batch but this is intentional  The TLDR answer is that if we allow ourselves to look at the norm of each gradient, we would know when a batch contains an outlier. Even if we do eventually clip it, this knowledge is a further source of information leak that would have to be addressed by our analysis.
If it’s helpful, let’s look at this from another angle: our guarantees will always have to be pessimistic, because by nature we need to protect the privacy of every sample, therefore we gotta make sure that whatever we do, we protect the privacy of each and every outlier. Let’s consider the DP definition and go back to the canonical example of an adversary that sees some outputs and needs to understand if they are seeing dataset D or dataset D’ (prime). They differ by a single example, so let’s say all examples are “easy” except from that one, which gets a huge gradient. This procedure of looking first and clipping later would output the same model, but now an adversary that sees the training loop as it trains would know who’s the outlier and beat us at the D vs D’ game.
That being said, this is potentially one of the things that we could do if we relaxed our guarantees by making less pessimistic assumptions on the capabilities of our adversary. It would be interesting to think about it, but to my knowledge it’s never been done


Question: Is there any literature/documentation on the necessity of cryptographically-secure RNG in Opacus? As far as I can tell it’s only used for noise generation so would like to understand the underlying risks/compromises. It looks to me like TF-Privacy isn’t using a CSPRNG for its noise generation so am curious about the reasoning for the divergence.

Context:
Excellent question! There are two separate uses of CSPRNG in Opacus - for batch composition (shuffling or sampling) and for noise generation. If batches are formed using an insecure RNG, the algorithm is still differentially private, but we don’t get to claim amplification by subsampling.

It looks to me like TF-Privacy isn’t using a CSPRNG for its noise generation so am curious about the reasoning for the divergence.

It’s a pragmatic choice. Basically, TF does not natively support CSPRNG; Pytorch’s decision to implement it is in no small part due to its commitment to supporting Opacus.
I’ll add parenthetically, that, strictly speaking, DP is an information-theoretic notion. As such, it cannot be satisfied with “just” cryptographically secure PRNG. For purists (or complexity theorists) out there, Opacus meets the definition of Computational Differential Privacy. In practice, all randomness available on computers is computational, and I don’t believe drawing the distinction is productive at this point of technology development.