akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Custom optimizer got an empty parameter list

ZeroMax: <p>Hello, new here. I am trying to create a custom optimizer in PyTorch, where the backprop takes place in a meta RL policy. However, I am seeing the above error. My models work fine on Adam and SGD, but not my optimizer.</p>
<p>Code:</p>
<pre><code class="lang-auto">class MetaBackProp(torch.optim.Optimizer):
	def __init__(self, params):
		
		self.param_shape_list = np.array([])
		for param in list(params):
			np.append(self.param_shape_list, list(param.size()))

		pseudo_lr = 1e-4
		pseudo_defaults = dict(lr=pseudo_lr)
		length = 100 #TODO: get shape, flatten, multiply...
		self.policy = AEPolicy(length)
		self.policy_optim = torch.optim.Adam(self.policy.parameters(), lr=pseudo_lr)
		super(MetaBackProp, self).__init__(params, pseudo_defaults)

	def step(self, closure=None):
		params = torch.cat([p.view(-1) for p in self.param_groups])
		self.policy_optim.zero_grad()
		quit()
</code></pre>
<p>Traceback:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "main.py", line 6, in &lt;module&gt;
    gan = CycleGAN()
  File "/home/ai/Projects_v2/R/cycle_gan.py", line 32, in __init__
    self.discriminator2_optim = MetaBackProp(self.discriminator2.parameters())
  File "/home/ai/Projects_v2/R/lr_schedule.py", line 34, in __init__
    super(MetaBackProp, self).__init__(params, pseudo_defaults)
  File "/home/ai/anaconda3/lib/python3.7/site-packages/torch/optim/optimizer.py", line 46, in __init__
    raise ValueError("optimizer got an empty parameter list")
ValueError: optimizer got an empty parameter list
</code></pre>

JuanFMontesinos: <p>Once you call list(params) you are exhausting the generator. Thus, when you call super init, it’s empty.</p>
Hyperparameter Search for Connect Four (DDQN)

spadel: <p>Hey guys, I just wrote a code to train a DDQN to learn Connect Four. As this is my first experience with DDQN’s, I have no ideas about which hyperparameters could roughly work out, if I train the network long enough. My convolutional network looks as follows:</p>
<pre><code class="lang-auto">class CNN(nn.Module):
    def __init__(self):
        # process 6x7
        super(CNN, self).__init__()
        self.process_cnn = nn.Sequential(
            # transforms to 5x6 (formula: out = (in-k+2*p)/s + 1)
            nn.Conv2d(1, 16, 4, stride=1, padding=1),
            nn.ReLU(True),
            # transforms to 4x5
            nn.Conv2d(16, 32, 3, stride=1, padding=1),
            nn.ReLU(True)
        )
        self.process_lin = nn.Sequential(
            nn.Linear(5 * 6 * 32, 64),
            nn.ReLU(True),
            nn.Linear(64, 7)
        )

    def process(self, x):
        # Apply convolutions
        x = self.process_cnn(x)
        # Flatten
        x = x.view([x.size(0), -1])
        # Apply linear layers
        x = self.process_lin(x)
        return x

    def forward(self, x):
        x = self.process(x)
        return x
</code></pre>
<p>From my experience, the number and size of the kernel is not too important to find convergence, as long as you can cover the complexity of the problem. What I am interested in specifically is, what batch size should I use and after how many batches should I update the target network. Also some approximate values for learning rate, the epsilon value and the discount would be very helpful. Currently I am trying out linearly decaying epsilon values and learning rates - is this in general a good idea?</p>
<p>Thank you very much!!</p>
Out of memory while using nn.DataParallel() but fits with one GPU

assyl_coop: <p>I have been implementing Deep Autoencoders for recommender systems.<br>
My data consists of users:535000 and items:1551731.</p>
<p>Model’s arichitecture is:<br>
############# AutoEncoder Model: #####################<br>
DeepEncoder(<br>
(en1): Linear(in_features=1551731, out_features=128, bias=True)<br>
(en2): Linear(in_features=128, out_features=192, bias=True)<br>
(en3): Linear(in_features=192, out_features=256, bias=True)<br>
(en4): Linear(in_features=256, out_features=320, bias=True)<br>
(dp1): Dropout(p=0.8)<br>
(su): SELU()<br>
(de1): Linear(in_features=320, out_features=256, bias=True)<br>
(de2): Linear(in_features=256, out_features=192, bias=True)<br>
(de3): Linear(in_features=192, out_features=128, bias=True)<br>
(de4): Linear(in_features=128, out_features=1551731, bias=True)<br>
)<br>
######################################################</p>
<p>If I use:<br>
rencoder=DeepEncoder(1551731)<br>
device=torch.device('cuda:0,1; if torch.cuda.is_available() else ‘cpu’)<br>
rencoder=nn.DataParallel(rencoder)<br>
rencoder.to(device)<br>
I am out of cuda memory/</p>
<p>But with the same model but keeping everything in one gpu works fine.<br>
Help please. I wanted to use both gpu in order to enhace my model architecture.</p>

Usman_Mahmood: <p>I have the same issue, did you find a fix?</p>
Torch > ONNX dynamic shape values

boriskourt: <p>I’ve trained a style transfer model based on <a href="https://github.com/rrmina/fast-neural-style-pytorch" rel="nofollow noopener">this implementation</a>.</p>
<p>As far as I can tell the following defines the shape of their input and model:</p>
<pre><code class="lang-python">    # Get Style Features
    imagenet_neg_mean = torch.tensor([-103.939, -116.779, -123.68], dtype=torch.float32).reshape(1,3,1,1).to(device)
    style_image = utils.load_image(STYLE_IMAGE_PATH)
    style_tensor = utils.itot(style_image).to(device)
    style_tensor = style_tensor.add(imagenet_neg_mean)
    B, C, H, W = style_tensor.shape
    style_features = VGG(style_tensor.expand([BATCH_SIZE, C, H, W]))
    style_gram = {}
    for key, value in style_features.items():
        style_gram[key] = utils.gram(value)
</code></pre>
<p>The trained model works on any shape of image input.</p>
<p>I’ve been able to successfully transfer this to ONNX, and use it from there.</p>
<p>Unfortunately only with static H and W using the following code:</p>
<pre><code class="lang-auto">    prior = torch.load("pth/" + name + ".pth");
    model = transformer.TransformerNetwork()     # [0] 
    model.load_state_dict(prior)

    dummy_input = torch.randn(1, 3, 640, 480).   # 640 x 480 is an arbitrary static H W

    torch.onnx.export(model, dummy_input, "onnx/" + name + ".onnx", verbose=True,
            input_names=["input_image"],
            output_names=["stylized_image"])
</code></pre>
<p>[0]: <a href="https://github.com/rrmina/fast-neural-style-pytorch/blob/master/transformer.py#L4" rel="nofollow noopener">transformer.py</a></p>
<p>I tried to use:</p>
<pre><code class="lang-auto">dynamic_axes={'input_image': {2:'height', 3:'width'}, 'stylized_image': {2:'height',  3:'width'}}
</code></pre>
<p>Both labeled and unlabeled.</p>
<p>Unfortunately I don’t know what to change the: <code>dummy_input = torch.randn(1, 3, 640, 480)</code> values to, or if they should remain as they are.</p>
<p>What would be the proper method for describing this model in order to get a good ONNX conversion?</p>
<p>Thanks!</p>

boriskourt: <p>As I don’t see a way to edit anymore I’ve posted a related question to SO: <a href="https://stackoverflow.com/questions/61231340/input-input-image-of-layer-63-not-found-in-any-of-the-outputs-of-the-preceed" rel="nofollow noopener">https://stackoverflow.com/questions/61231340/input-input-image-of-layer-63-not-found-in-any-of-the-outputs-of-the-preceed</a></p>
<p>In case someone stumbles over this via search.</p>
Pytorch C++ Front End

Jay_Fartiyal: <p>Hi,</p>
<p>I have to deploy object detection model in Android. Currently, it is taking lot of time 150-250ms. And this is just the model inference speed. I have used MobileNetV1 as the backbone with 0.25 width multiplier.</p>
<p>Now, I am looking for reducing latency and thinking to use Pytorch C++ Frontend.</p>
<p>Does anyone knows, how fast is Pytorch C++ Frontend or is it same as Pytorch python.</p>
<p>Another option is quantisation. Does it provide a major boost in speed.</p>
<p>I am really having high hopes on Pytorch C++ Frontend but not getting much information from the forum</p>
How to make dataset from 1 image

kelam_goutam: <p>I am given a task to locate the defect in images. However, they have just given only 1 sample each for a good part and bad part. I am not understanding how to make the dataset for this. <img src="https://discuss.pytorch.org/uploads/default/original/3X/8/1/81fd0a0575f8818d938640b65d108e3e2a5ec1a7.jpeg" alt="IMG-20200317-WA0047" data-base62-sha1="ixVJ6Jl1Zx5u5IwDn4Kx2mA9nNR" width="640" height="480"> <img src="https://discuss.pytorch.org/uploads/default/original/3X/e/7/e7825ae59403bf536a2b152a440570643ed876ec.jpeg" alt="IMG-20200317-WA0046" data-base62-sha1="x21zXXkXWnKckUop74YuD74YyxC" width="640" height="480"></p>
<p>The image with 2 plain holes is bad part where as the other is a good part.</p>

ptrblck: <p>Are you sure you have only a single input image for both classes without any validation and test set?<br>
How will your model be evaulated?<br>
The task seems to be a bit “weird”.</p>

braindotai: <p>This task for sure is a bit “weird”. There should be no reason why you can’t get enough images, it just a matter of few more camera clicks, in fact you record a video and then extract frame by frame images to create your dataset.<br>
In any case all above is not possible then there is only one method left, you can use massive data augmentation on that single image. More about data transformation <a href="https://pytorch.org/docs/stable/torchvision/transforms.html" rel="nofollow noopener">here</a>.</p>

ruculent: <p>What he means is that he has only 1 image for the image segmentation task. Perhaps OP wants you could point to any source where more images can be obtained</p>
Working with synthetic images and real ones

thejoe: <p>Hi everyone,</p>
<p>I’m working with a resnet50 model to make some predictions on some complex images. I have also a consistent quantity of synthetic images similar the real ones. Problem is that my model has to be trained only on synthetic images and be evaluated on real images. During training phase I get low validation accuracy (around 10%). So, my question is if there exists some techniques to convert one particular real image into a synthetic one or viceversa? thank you.</p>

jmandivarapu1: <aside class="quote no-group" data-username="thejoe" data-post="1" data-topic="76869">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/thejoe/48/11809_2.png" class="avatar"> thejoe:</div>
<blockquote>
<p>During training phase I get low validation accuracy (around 10%). So, my question is if there exists some techniques to convert one particular real image into a synthetic one or viceversa? thank you.</p>
</blockquote>
</aside>
<p>I know this is a open ended question. Here is my thoughts</p>
<ol>
<li>
<p>First check how similar is the real images to synthetic ones either using some similarity measure. Because if it is not similar enough then the model is training on one distribution and testing on another distribution which is non IID assumption. So your accuracy will always be low.</p>
</li>
<li>
<p>Secondly converting real to  synthetic can be done using multiple techniques out there like GAN or regular methods too. But the question want in the synthetic images is it just the pixel changes or noise …etc</p>
</li>
</ol>
Compilation PyTorch

Snuffy4: <p>Please tell me how to compile PyTorch in Google Colab, and then install it in Anaconda 3 on my computer?</p>

jmandivarapu1: <p><code>pip install torch torchvision</code></p>
<p>I assume google colab coes with pytorch installed if not u can just install it with <code>!pip install torch torchvision</code> in jupyter notebook cell</p>
Padding tensors efficiently?

Felix_Labelle: <p>Is there an efficient way of padding a list of 3d tensors? The function below takes ~2.5 seconds to pad 50 tensors (L x 36 x 2048), where L varies between [3,7].</p>
<pre><code class="lang-auto">    def padFeats(tensors_to_pad, batch_size, pad_val):
        max_len = max([len(sent) for sent in tensors_to_pad])
        padded_tensor = torch.ones((batch_size,max_len,36,2048)) * pad_val
        pad_start = time.perf_counter()
        for i, val in enumerate(tensors_to_pad):
            padded_tensor[i,:len(val), ...] = torch.tensor(val)
        print("{} tensors were padded in {}".format(len(tensor_to_pad),pad_start - time.perf_counter()))
        return padded_tensor
</code></pre>
Reuse weights from another neural network

Bob_Van_Der_Meer: <p>Hi all,</p>
<p>I just built and trained a neural network. The neural network consists of an imput image, that is linearly scaled to a hidden layer with N hidden units. The results of the hidden layer are then once again linearly saled to the output layer. In both the hidden and output layer i’'m using ReLu activation function. However now i want to create a second network, which has a similar form as the previous but this time the hidden layer needs to consist of N+1 hidden units. Furthermore i would like to reuse the weights found from the previous network, and initialize the extra needed weight using a normal distribution. The net i have so far looks like this. Note, the idea is to extend this to a larger network, for the first initialization i want to use the xavier numbers. I hope someone can give some general advice.</p>
<pre><code class="lang-auto">class Neural_Network(nn.Module):
    def __init__(self, inpsize, outsize, hidsize, N, n, Weights1,Weights2):
        super(Neural_Network, self).__init__()
        self.inputSize = inpsize
        self.outputSize = outsize
        self.hiddenSize = hidsize

        if N == 0:     
            self.W1 = nn.Linear(self.inputSize,self.hiddenSize, bias=False)
            self.W2 = nn.Linear(self.hiddenSize,self.inputSize, bias=False)
        
            torch.nn.init.xavier_uniform_(self.W1.weight, gain=1.0)
            torch.nn.init.xavier_uniform_(self.W2.weight, gain=1.0)
        else:
            self.W1 = nn.Linear(self.inputSize,self.hiddenSize, bias=False)
            self.W2 = nn.Linear(self.hiddenSize,self.inputSize, bias=False)
            
    def forward(self, X):
        x = F.relu(self.W1(X))
        x = F.relu(self.W2(x))
        return x, self.W1.weight, self.W2.weight
      
</code></pre>

imbibekk: <p><a href="https://discuss.pytorch.org/t/check-if-a-module-is-in-the-state-dict-or-not/26078/2">This thred</a> is what you maybe looking for. It shows how you can filter-out trained weights from one model to another</p>
Pytorch.cuda.is is avalaible false nvidia-sim not recognized

Aurelie_Sotura: <p>Hello</p>
<p>I want to use pytorch but I get pytorch.cud is.avalaible  FALSE. I tried to install CUDA but I don’t think/know if it worked.</p>
<p>Here are my configuration:<br>
Python 3.7.3<br>
Nvidia Getforce GTX 960M<br>
I am using pip.</p>
<p>Here is what I did.<br>
I installed Visual Studio 2019. I installed CUDA Toolkit 10.2 Download (but I am not sure it worked though) using the following link: <a href="https://developer.nvidia.com/cuda-downloads?target_os=Windows&amp;target_arch=x86_64&amp;target_version=81&amp;target_type=exelocal" rel="nofollow noopener">https://developer.nvidia.com/cuda-downloads?target_os=Windows&amp;target_arch=x86_64&amp;target_version=81&amp;target_type=exelocal</a></p>
<p>When I run on my command, I get:<br>
(python) torch.cuda.is_available()<br>
false</p>
<ul>
<li>(python) torch.backends.cudnn.enabled<br>
true</li>
<li>(cmd) nvidia-smi<br>
nvidia not defined</li>
<li>(cmd) nvcc --version<br>
nvcc not defined</li>
</ul>
<p>I looked up many wepages but I didn’t find something to solve my problem. Do you think I should use Anaconda instead of pip? Can it be a matter of path?</p>
<p>Thanks a lot for your patience.</p>

peterjc123: <p>What is the output of <code>"C:\Program Files\NVIDIA Corporation\NVSMI\nvidia-smi.exe"</code> in CMD?</p>

Aurelie_Sotura: <p>Thank you for your answer. It says it is not recognized as an intern command (in French).</p>

peterjc123: <p>So looks like you don’t have the GPU driver installed.</p>
Remove CNN Layers in Resnet18 for Transfer Learning

irish_guy: <p>I am looking to do some experiments in transfer learning using Resnet18. Currently I replace the fully connected layer as follows :</p>
<p>fc_inputs = model.fc.in_features<br>
model.fc = nn.Sequential(nn.Linear(fc_inputs, num_classes),)</p>
<p>I am trying to figure out how to remove conv5_x and connect the fully connected layer to the conv4_x layer. ( Or connect it to conv3_x )</p>
<p>I want to compare the results tapping into the network at different places.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://forums.fast.ai/uploads/default/original/3X/f/9/f9a8eaaa866c2b783162fe65afdff67171e993fb.png" title="" rel="nofollow noopener"><img src="https://forums.fast.ai/uploads/default/original/3X/f/9/f9a8eaaa866c2b783162fe65afdff67171e993fb.png" alt width="553" height="500"><div class="meta">
<svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use xlink:href="#far-image"></use></svg><span class="filename"></span><span class="informations">794×717</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use xlink:href="#discourse-expand"></use></svg>
</div></a></div></p>

AladdinPerzon: <p>If we import the ResNet model <code>conv5_x</code> will be <code>layer4</code>. We could do the following:</p>
<pre><code class="lang-auto">model = torchvision.models.resnet18(pretrained=True)
model.layer4 = nn.Identity()
model.fc = nn.Linear(256, num_classes)
</code></pre>
<p>And you could use on the <code>model.fc = nn.Sequential(nn.Linear(), ...)</code> if you wish to write several linear layers.</p>

irish_guy: <p>Hi Aladdin,<br>
Thanks for your reply and and for the transfer learning it looks like it would work great. I will test it.</p>
<p>Is there a way to completely remove the conv4_x layer from the model?</p>
Best practice on saving model params together with metrics

carlstronaut: <p>Hello,</p>
<p>I’m just wondering what the best practice is regarding storing models/model params together with e.g. classification metrics for that particular model.</p>

ptrblck: <p>The recommended way is storing the <code>state_dicts</code> as described in the <a href="https://pytorch.org/docs/stable/notes/serialization.html">serialization docs</a>.<br>
Often a <code>dict</code> is stored with all <code>state_dicts</code> and other attributes as shown in the <a href="https://github.com/pytorch/examples/blob/391be73bbd191a08776c9ba05d29ef1e05c0c83b/imagenet/main.py#L254-L260">ImageNet example</a>.</p>

carlstronaut: <p>Just what I needed, thanks!</p>
CTC loss support PackedSequence as input?

Legend: <p>When I pass a batch of sequences of variable lengths to torch.nn.CTCLoss, I need to first run pad_packed_sequence() to pad all the sequences to the maximun sequence length. Because CTCLoss only accept Tensor of size TNC(or NTC) as input.</p>
<p>But the problem is my sequence lengths is heavily biased, for an extrem example my sequence lengths is like [10000, 1, 1, 1, 1 … 1]. In this situation the padding cost much time and memory.</p>
<p>So can I just pass a PackedSequence to CTCLoss and skip pad_packed_sequence()?</p>
Puzzling question

youi: <p>I have code that consistently achieves a certain score on a computer vision task.</p>
<p>Whenever I grab that code and try to make a class out of it with a .run() and similar methods, I get a slightly lower score. Even when I average over many runs.</p>
<p>Same exact code, but I try to abstract it. I copy paste it into methods basically. I do not change anything. I make attributes out of model, optimizer, scheduler and use those attributes in methods.</p>
<p>Is there anything that would be causing this from PyTorch?</p>
Attention in many-to-one LSTM

dlarah: <p>Hi folks,</p>
<p>I have read a lot about attention mechanisms in Encoder-Decoder networks.<br>
All examples I’ve found have an Encoder -&gt; Attention -&gt; Decoder Mechanism.</p>
<p>My LSTM which I use for next class prediction (input is a sequence of 10 concatenated Bert-embeddings, so n_input=10 * 768) (more precisely I’m trying to do anomaly detection). It works, I’m getting results which are “ok”, recall is 1.0, but F1 Score of “only” 0.55. So I was thinking of implementing an attention mechanism.</p>
<pre><code>def __init__(self, n_input, n_hidden_units, n_layers, n_classes, tie_weights=False, train_mode=False):
    super(LSTM, self).__init__()
    self.n_hidden_units = n_hidden_units
    self.n_layers = n_layers
    self.train_mode = train_mode

    # Layers

    self.attn = nn.Linear(self.n_hidden_units * 2, n_input)
    self.attn_combine = nn.Linear(self.hidden_size * 2, self.hidden_size)
    self.lstm = nn.LSTM(input_size=n_input, hidden_size=n_hidden_units, num_layers=n_layers,
                        dropout=0.2, batch_first=True)
    self.decoder = nn.Linear(n_hidden_units, n_classes)

    if tie_weights:
        if n_input != n_hidden_units:
            raise ValueError('When using the tied flag, nhid must be equal to emsize')
        self.decoder.weight = self.encoder.weight

    self.init_weights()

def init_weights(self):
    initrange = 0.1
    self.decoder.bias.data.zero_()
    self.decoder.weight.data.uniform_(-initrange, initrange)

def forward(self, input, hidden):
    output, hidden = self.lstm(input, hidden)
    if self.train_mode: output = nn.Dropout(p=0.1)(output)

    decoded = self.decoder(output[:, -1, :])
    log_props = F.log_softmax(decoded, dim=1)
    return log_props, hidden

def init_hidden(self, bsz, device):
    weight = next(self.parameters())
    return (weight.new_zeros(self.n_layers, bsz, self.n_hidden_units).to(device),
            weight.new_zeros(self.n_layers, bsz, self.n_hidden_units).to(device))
</code></pre>
<p>I know from the examples, that the attention layers have to be put somewhere before the lstm. All the examples use many-to-many, but I’m doing many-to-one, and I don’t just want to put the layers and make it work somehow, without knowing exactly how it’s done right.<br>
So if someone could show me where in the code I would have to put what exactly, that would be very helpful.</p>

vdw: <p>You can have a look at my implementation of a RNN classifier (GRU or LSTM) with or without attention here: <a href="https://github.com/chrisvdweth/ml-toolkit/blob/master/pytorch/models/text/classifier/rnn.py" rel="nofollow noopener">Github repository</a>.</p>
<p>It basically directly derives from a many-to-many model, just with many=1 for the decoder. So there’s for example no loop in the decoder that generates more and more outputs (e.g., for machine translation).</p>
<p>I hope that helps.</p>
Loss Function in Multi-GPUs training (PyTorch)

al-yakubovich: <p>I use Pytorch and BERT to traing a model. Everithing works great on one GPU, but when I try to use multi GPUs I am getting an error:</p>
<pre><code>ValueError                                Traceback (most recent call last)
&lt;ipython-input-168-507223f9879c&gt; in &lt;module&gt;()
     92         # single value; the `.item()` function just returns the Python value
     93         # from the tensor.
---&gt; 94         total_loss += loss.item()
     95 
     96         # Perform a backward pass to calculate the gradients.

ValueError: only one element tensors can be converted to Python scalars
</code></pre>
<p>Can someone help me what I am missing and how I should fix it?</p>
<p>Here is my code for training:</p>
<pre><code>import random
seed_val = 42
random.seed(seed_val)
np.random.seed(seed_val)
torch.manual_seed(seed_val)
torch.cuda.manual_seed_all(seed_val)
loss_values = []
for epoch_i in range(0, epochs):
    
    t0 = time.time()
    total_loss = 0
    for step, batch in enumerate(train_dataloader):
        if step % 40 == 0 and not step == 0:
            elapsed = format_time(time.time() - t0)
            
        b_input_ids = batch[0].to(device).long() 
        b_input_mask = batch[1].to(device).long()
        b_labels = batch[2].to(device).long()
        model.zero_grad()        
        outputs = model(b_input_ids, 
                    token_type_ids=None, 
                    attention_mask=b_input_mask, 
                    labels=b_labels)
       
        loss = outputs[0]
        total_loss += loss.item()
        loss.backward()
        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        optimizer.step()
        scheduler.step()
    avg_train_loss = total_loss / len(train_dataloader)            
    loss_values.append(avg_train_loss)
    print("")
    print("  Average training loss: {0:.2f}".format(avg_train_loss))
    print("  Training epcoh took: {:}".format(format_time(time.time() - t0)))
</code></pre>
<p>And here is my code for the model:</p>
<pre><code>from transformers import BertForSequenceClassification, AdamW, BertConfig
model_to_parallel = BertForSequenceClassification.from_pretrained(
    "./bert_cache.zip", 
    num_labels = 2, 
    output_attentions = False,
    output_hidden_states = False,
)
model = nn.DataParallel(model_to_parallel,  device_ids=[0,1,2,3]) 
model.to(device)</code></pre>

ptrblck: <p>Could you print the shape of <code>outputs</code> and <code>loss</code> before the error is raised?<br>
Apparently you are not getting a scalar loss value and I’m also unsure why you are indexing <code>output</code> at index0.</p>

al-yakubovich: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Thanks!  I got it. After loss <code>loss = outputs[0]</code> the <code>loss</code> is a multi-element tensor, the size is number of GPUs. I need to use <code>loss = outputs[0].mean()</code> instead.</p>
Freeze last layers of the model

lanhhv84: <p>Hi everyone,</p>
<p>I am trying to implement VGG perceptual loss in pytorch and I have some problems with autograd. Specificly, the output of my network (1) will to through VGG net (2) to calculate features. Also, my ground truth images also go through VGG net to calculate features too. After that, I want to calculate loss based on these features. However, I think that the VGG net is pretrained and it shouldn’t update its gradient. I try to set require_grad = False for every layer in the VGG net and I get an error before training “element 0 of tensors does not require grad and does not have a grad_fn”. I suspect that freezing last layers in the network prevent other layers from calculating gradients too. Is there anyway to bypass it or Should I just let the VGG train with other parameters. Thank all for helps.</p>

ptrblck: <p>If you are setting <code>requires_grad = False</code> for all parameters, the error message is expected, as Autograd won’t be able to calculate any gradients, since no parameter requires them.</p>
<p>Could you describe your use case a bit, i.e. which layers would you like to train and which should be frozen?</p>

lanhhv84: <p>I have an generator with grad that translate an image into different image of same shape. I pass the output of the generator to vggnet(requires_grad=False) (1) and I also pass ground truth image into vggnet(requires_grad=False) (2). At the end, I want to calculate l1 loss between (1) and (2). Thanks.</p>

ptrblck: <p>This workflow should work as shown in this dummy example:</p>
<pre><code class="lang-python"># Setup
generator = nn.Conv2d(3, 3, 3, 1, 1)
dummy = nn.Conv2d(3, 3, 3, 1, 1)

data = torch.randn(1, 3, 24, 24)
target = torch.randn(1, 3, 24, 24)

# Freeze dummy model
for param in dummy.parameters():
    param.requires_grad_(False)

# Forward pass for data
out = generator(data)
out = dummy(out)

# Forward pass for target
target = dummy(target)

# Loss calculation
loss = ((out - target)**2).sqrt().mean()
loss.backward()

# Check gradients
print(generator.weight.grad)
print(dummy.weight.grad)
</code></pre>
<p>As you can see, <code>generator</code> will get valid gradients, while <code>dummy</code> won’t.</p>
<p>Could you post a code snippet to reproduce this issue, so that we could have a look?</p>
Device issue when creating a custom activation function

shamoons: <p>I have a custom activation function:</p>
<pre><code class="lang-auto">import torch
import math

class CustAct(torch.nn.Module):
    def __init__(self, degree=4, alphas=None):
        super(CustAct, self).__init__()
        if alphas:
            self.alphas = torch.nn.Parameter(torch.Tensor(alphas))
        else:
            self.alphas = torch.nn.Parameter(torch.ones(degree + 1))

        self.alphas.requiresGrad = True  # set requiresGrad to true!
        self.beta.requiresGrad = True  # set requiresGrad to true!

    def forward(self, x):
        val = torch.zeros(x.size())
        for d in range(len(self.alphas)):
            val += self.alphas[d] * x

        return val
</code></pre>
<p>And in my model:</p>
<pre><code class="lang-auto">self.linear = torch.nn.Sequential(
                torch.nn.Linear(feature_dim, feature_dim),
                CustAct(alphas=[1, 2, 3, 4]),
                torch.nn.Linear(feature_dim, feature_dim),
                torch.nn.ReLU6()

            )
</code></pre>
<p>I am running on a GPU, but when I try with the custom activation function, I get:</p>
<pre><code class="lang-auto">    val += self.alphas[d] * x
RuntimeError: expected device cpu but got device cuda:0
</code></pre>
<p>I’m not sure why it expects a <code>cpu</code>.</p>

imbibekk: <p>Try this!</p>
<pre><code class="lang-auto">import torch
import math

class CustAct(torch.nn.Module):
    def __init__(self, degree=4, alphas=None):
        super(CustAct, self).__init__()
        if alphas:
            self.alphas = torch.nn.Parameter(torch.Tensor(alphas))
        else:
            self.alphas = torch.nn.Parameter(torch.ones(degree + 1))

        self.alphas.requiresGrad = True  # set requiresGrad to true!
        #self.beta.requiresGrad = True  # set requiresGrad to true!

    def forward(self, x):
        val = torch.zeros(x.size()).cuda()  # move to GPU
        for d in range(len(self.alphas)):
            val += self.alphas[d] * x

        return val

feature_dim = 128
linear = torch.nn.Sequential(
                torch.nn.Linear(feature_dim, feature_dim),
                CustAct(alphas=[1, 2, 3, 4]),
                torch.nn.Linear(feature_dim, feature_dim),
                torch.nn.ReLU6()

            )



linear.cuda() 

rand_inp = torch.randn((4, 128)).cuda()
out = linear(rand_inp)
print(out.shape)  # torch.Size([4, 128])

</code></pre>
When using a scheduler should we pass val_acc or val_loss

alx: <p>For example, ReduceLROnPlateau(val_loss) vs ReduceLROnPlateau(val_acc). Its confusing since in some epochs loss increases and accuracy increases. They’re not 100% correlated on every epochs.</p>

ptrblck: <p>This might be the case and you could adapt the <code>patience</code> value for it.<br>
E.g. if you current training sees a lot of these stats, you might want to use a higher patience to avoid decreasing the learning rate too early.</p>
How and when will GPU be faster than CPU? Only when the batch_size of the tensor > 1?

crissallan: <p>Hello,<br>
I was recently doing some data processing by Torch and met an interesting problem. The whole data is only a 300 * 3 tensor，and I applied some matrix multiplication(dot product, torch.cross…etc.) on this data tensor.</p>
<p>I tried on both CPU and GPU and find the comsumed time of them are almost the same(GPU is slower even, I think it is because I need to move the data from CPU to GPU).</p>
<p>So, I am very curious about that when will the GPU faster than CPU.I readed some concepts about paralle computation, but haven’t figure it out. Does paralle computation means that the GPU only parallelly process tensors in one batch, and it won’t parallely do the broadcasting computation？ So only when my data tensor with shape: batch_size * 300 * 3, where batch_size &gt; 1, the GPU computation will be faster than CPU?</p>
<p>Many thanks!</p>

ptrblck: <p>Generally, you might not see an expected speedup on the GPU, if you are using a lot of “small” operations and are thus limited by the memory bandwidth.</p>
<p>That being said, even with a batch size of 1 you would see a speedup, if the computational workload of the model is not tiny.</p>
<p>Could you post your model definition so that we could have a look at it, please?</p>
Runing speeds of a model with difficult weights are quite different,

iminfine: <p>I trained a model twice. As I tested the two best models, the FPS are around 8 and 2 respectively. Nothing else was changed except the weights of the model.  How could this happen?</p>

ptrblck: <p>Could you explain your profiling and use case a bit?<br>
Also, are you using the same input shapes?<br>
Is the model using the CPU or GPU? In the former case, you might see a performance hit due to the processing of denormal values, which is slower on the CPU. In that case, try to set <code>torch.set_flush_denormal(True)</code> and rerun the code. If you are using the GPU, make sure to synchronize the code properly via <code>torch.cuda.synchronize()</code> before starting and stopping the timer.</p>
Freezing some parameters in a layer

TinfoilHat0: <p>Hi,</p>
<p>I’d like to freeze a set of selected parameters in a given layer when I’m training a neural network.<br>
How’s this done in PyTorch?</p>

ptrblck: <p>Have a look at <a href="https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html#set-model-parameters-requires-grad-attribute">this tutorial</a>.</p>

TinfoilHat0: <p>This seems to be freezing weights layer-wase, i.e., all the params of a layer are frozen.</p>
<p>What I want is something more fine-tuned. For example, I’d like to identify top 5% of parameters (by their weights) in a given layer and only freeze those.</p>

ptrblck: <p>In this case you won’t be able to use the <code>requires_grad</code> attribute, as it’s defined for the whole parameter.<br>
However, you could zero out the gradients of the specific values which shouldn’t be updated using a hook via <code>param.register_hook</code>.<br>
Note that even with a zero gradient your parameters might still be updated, e.g. if you are using weight decay for all parameters.</p>
Other functions CNN

jasg: <p>that other functions apart from linear and log_softmax exist in pytorch,<br>
How to configure in this function:</p>
<pre><code class="lang-auto">class ConvolutionalNetwork(nn.Module):
        def __init__(self):
            super().__init__()
            self.conv1 = nn.Conv2d(tipo_image, 6, kernel_size=(kernel,kernel), stride=stridee,padding=paddingg,bias=biass)
            self.conv2 = nn.Conv2d(6, 16, kernel_size=(kernel,kernel), stride=stridee,padding=paddingg,bias=biass)
            self.conv3 = nn.Conv2d(16, 32, kernel_size=(kernel,kernel), stride=stridee,padding=paddingg,bias=biass)

        
            self.fc1 = nn.Linear(f*f*32, 120)
            self.fc2 = nn.Linear(120, 84)
            self.fc3 = nn.Linear(84, 18)

        def forward(self, X):
            X = F.relu(self.conv1(X))
            X = F.max_pool2d(X, 2, 2)
            X = F.relu(self.conv2(X))
            X = F.max_pool2d(X, 2, 2)
            X = F.relu(self.conv3(X))
            X = F.max_pool2d(X, 2, 2)
            X = X.view(-1, f*f*32)
            X = F.relu(self.fc1(X))
            X = F.relu(self.fc2(X))
            X = self.fc3(X)
            return F.log_softmax(X, dim=1)
    torch.manual_seed(101)
    CNNmodel = ConvolutionalNetwork()
    criterion = nn.CrossEntropyLoss()
    optimizer = torch.optim.Adam(CNNmodel.parameters(), lr=0.001)
    print(CNNmodel)</code></pre>

mohaimen: <p><a class="mention" href="/u/jasg">@jasg</a> Not sure if I understood your question. Could you please clarify it for me?</p>

jasg: <p>ok, apart from the nn.linear and softmax function that other alternatives I can use in the class ConvolutionalNetwork</p>

mohaimen: <p>This is the list of all Non-Linear activation functions available in pytorch:<br>
<a href="https://pytorch.org/docs/stable/nn.functional.html#non-linear-activation-functions" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.functional.html#non-linear-activation-functions</a>.</p>
<p>I am sure you know this but just reiterating that we commonly use sigmoid(binary classification) and softmax/logsoftmax (multi-class).</p>
<p>Please let me know if you want more information.</p>

jasg: <p>I understand that since I have 18 classes I must use the softmax; but it is not clear to me what the other alternatives would be for the linear function, sorry this is new for me</p>

mohaimen: <p>I am not clear if you are just talking about nn.Linear now. If yes, this is basically adds a dense (fully connected) layer in your network.</p>

jasg: <p>I want to learn more about CNN, so I consult other options besides nn. Linear i can use</p>

mohaimen: <p>A typical cnn structure should be like this:<br>
input -&gt; convolution -&gt; pooling -&gt; convolution -&gt; pooling -&gt; fully connected (linear) -&gt; output(softmax/sigmoid)</p>
<p>a convolution layer usually consists of con-&gt;batch norm -&gt; non-linearity (e.g. relu)</p>
<p>Hope this helps.</p>
Some questions about the style-transfer implementaton

PKUFlyingPig: <p>I want to implement a style-transfer demo use a pertained VGG model.So how can I use the torchvision.model package which provides the pertained model for me and obtain the output of each activation layers (to compute the style-error)?Meantime how can I define the optimizer to optimize the Generated image in order to minimize the loss?</p>

kshitij: <p>This link should be helpful.</p>
<p><a href="https://pytorch.org/tutorials/advanced/neural_style_tutorial.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/advanced/neural_style_tutorial.html</a></p>

PKUFlyingPig: <p>Thank you so much ! I just discovered that there were so many good resources on the website you provided.</p>
Saving customized model architecture

marph91: <p>Hi,</p>
<p>I got a simple model with a given architecture. After training I applied quantization and added a custom quantization layer after each convolution layer.</p>
<p>Now I would need to save this architecture and access it from a different directory. My current way is to save it via <code>torch.save</code> and access it later via <code>torch.load</code>. The problem is that I have to keep the exact directory structure, as described in <a href="https://pytorch.org/docs/stable/notes/serialization.html#" rel="nofollow noopener">https://pytorch.org/docs/stable/notes/serialization.html#</a>, i. e. the file which includes the custom layer has to be carried everywhere the <code>torch.load</code> gets executed.</p>
<p>Is there another way to save the network architecture in pytorch? I’m aware of saving the weights via <code>statedict</code>, but I couldn’t find anything about saving the architecture.</p>

find1dream: <p>Save:</p>
<pre><code class="lang-auto">checkpoint = {'model': Classifier(),
          'state_dict': model.state_dict(),
          'optimizer' : optimizer.state_dict()}

torch.save(checkpoint, 'checkpoint.pth')
</code></pre>
<p>Load:</p>
<pre><code class="lang-auto">def load_checkpoint(filepath):
    checkpoint = torch.load(filepath)
    model = checkpoint['model']
    model.load_state_dict(checkpoint['state_dict'])
    for parameter in model.parameters():
        parameter.requires_grad = False

    model.eval()
    return model

model = load_checkpoint('checkpoint.pth')
</code></pre>

novioleo: <p>i think you’d like to use the static graph,eg. <code>proto</code> of caffe.<br>
however,the AFAIK pytorch have NOT imple this future.</p>

Anshumaan_Dash: <p>Hi there,<br>
This would solve the issue is one is importing the model in the same folder structure and environment. However, as of now, Pytorch doesn’t support saving the architecture.</p>