akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Parallel Gpu training - Multiple models, Same input data

z000: <p>Hi all!<br>
Is the following pipeline suitable to train at the same time two models in two different GPUs, but using the same input data to optimize the RAM usage?<br>
Or are there any “hidden” issues or data sharing/concurrency I am unaware of?</p>
<pre><code class="lang-python">elements = [list of input data loaded in RAM memory]

customDataset_valid = CustomDataset(elements[valid_from: valid_to])
customDataset_test = CustomDataset(elements[test_from: test_to])
customDataset_train = CustomDataset(elements[train_from: train_to])

loader_valid_1 = torch.utils.data.DataLoader(dataset=customDataset_valid, batch_size=BS, num_workers=0)
loader_test_1 = torch.utils.data.DataLoader(dataset=customDataset_test, num_workers=0)
loader_train_1 = torch.utils.data.DataLoader(dataset=customDataset_train, batch_size=BS num_workers=0)

loader_valid_2 = torch.utils.data.DataLoader(dataset=customDataset_valid, batch_size=BS, num_workers=0)
loader_test_2 = torch.utils.data.DataLoader(dataset=customDataset_test, num_workers=0)
loader_train_2 = torch.utils.data.DataLoader(dataset=customDataset_train, batch_size=BS num_workers=0)

net1 = NET_A()
net1 = net1.to(device_1, dtype=torch.float)
criterion1 = nn.CrossEntropyLoss().to(device_1, dtype=torch.float))
optimizer1 = optim.SGD(net1.parameters())
thread1 = threading.Thread(target=my_train, args=(device_1, criterion1, optimizer1, net1, loader_train_1, loader_valid_1))

net2 = NET_B()
net2 = net2.to(device_2, dtype=torch.float)
criterion2 = nn.CrossEntropyLoss().to(device_2, dtype=torch.float))
optimizer2 = optim.SGD(net2.parameters())
thread2 = threading.Thread(target=my_train, args=(device_2, criterion2, optimizer2, net2, loader_train_2, loader_valid_2))

thread1.start()
thread2.start()

thread1.join()
thread2.join()
</code></pre>
<p>my_train is a function that simply performs the training operations using the given arguments.</p>
<p>Thanks.</p>
How to sample minibatches without using for loop?

11130: <p>Hello, everyone! How can I define a function that takes a dataset and an index list as input and returns a tensor?</p>
<p>For example, given a dataset defined as below</p>
<pre><code class="lang-auto">import torch
import torch.utils.data
import torchvision.datasets as dset
import torchvision.transforms as transforms
import torchvision.utils as vutils
dataroot = "datasets/celebA/"
image_size = 64
dataset = dset.ImageFolder(root=dataroot,
                           transform=transforms.Compose([
                               transforms.Resize(image_size),
                               transforms.CenterCrop(image_size),
                               transforms.ToTensor(),
                               transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),
                           ]))
</code></pre>
<p>I want to define a function</p>
<pre><code class="lang-auto">def minibatch(dataset,indices):
    """

    :param dataset:
    :param indices: an index list
    :return: a N*C*H*W tensor that contains feature vectors corresponding to samples with these indices
    """
</code></pre>

JuanFMontesinos: <p>You can order dataloader indices as you want and use typicall pytorch dataloader. You will sequentially load batches with the mixtures that you want.</p>

11130: <p>I find out this is actually very easy <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"></p>
<pre><code class="lang-auto">
def batch(batch_size=batch_size, lower=0, upper=n_sample):
    indices = np.random.randint(low=lower, high=upper, size=batch_size)
    batch_data = torch.FloatTensor(batch_size, nc, image_size, image_size)
    for i in range(batch_size):
        sample, target = dataset[indices[i]]
        batch_data[i] = sample
    return batch_data

</code></pre>

eltoto1219: <p>Once you load the dataset, create a dataloader, and since the dataloader and then you can do something like:</p>
<pre><code class="lang-auto">a = iter(dataloader)
b = next(a)
</code></pre>
<p>where b is a minibatch.</p>
<p>An example:</p>
<pre><code class="lang-auto">    dataset = VQA_dataset(ROOT_DIR, train = False)


    loader = torch.utils.data.DataLoader(
                    dataset,
                    batch_size= 3,
                    shuffle=True,
                            )
    data = next(iter(loader))
    v, bb, spat, obs, a, q,  q_len, item = data


    torch.save(data, DUMMY_DATA)

</code></pre>
Pytorch retinanet - multi gpu issue

EliB: <p>Training runs on a server with a single gpu 1080ti however it isn’t running on  another workstation with 2 rtx 2080 ti nvlink with the error:</p>
<p>Traceback (most recent call last):<br>
File “”, line 1, in <br>
File “/opt/conda/lib/python3.6/multiprocessing/spawn.py”, line 105, in spawn_main<br>
exitcode = _main(fd)<br>
File “/opt/conda/lib/python3.6/multiprocessing/spawn.py”, line 114, in _main<br>
prepare(preparation_data)<br>
File “/opt/conda/lib/python3.6/multiprocessing/spawn.py”, line 225, in prepare<br>
_fixup_main_from_path(data[‘init_main_from_path’])<br>
File “/opt/conda/lib/python3.6/multiprocessing/spawn.py”, line 277, in _fixup_main_from_path<br>
run_name=“<strong>mp_main</strong>”)<br>
File “/opt/conda/lib/python3.6/runpy.py”, line 263, in run_path<br>
pkg_name=pkg_name, script_name=fname)<br>
File “/opt/conda/lib/python3.6/runpy.py”, line 96, in _run_module_code<br>
mod_name, mod_spec, pkg_name, script_name)<br>
File “/opt/conda/lib/python3.6/runpy.py”, line 85, in _run_code<br>
exec(code, run_globals)<br>
File “/home/elib/Dev/Retinanet_PT/UseExample_train.py”, line 39, in <br>
main.main(args)<br>
File “/home/elib/Dev/Retinanet_PT/retinanet/main.py”, line 185, in main<br>
torch.multiprocessing.spawn(worker, args=(args, world, model, state), nprocs=world)<br>
File “/opt/conda/lib/python3.6/site-packages/torch/multiprocessing/spawn.py”, line 158, in spawn<br>
process.start()<br>
File “/opt/conda/lib/python3.6/multiprocessing/process.py”, line 105, in start<br>
self._popen = self._Popen(self)<br>
File “/opt/conda/lib/python3.6/multiprocessing/context.py”, line 284, in _Popen<br>
return Popen(process_obj)<br>
File “/opt/conda/lib/python3.6/multiprocessing/popen_spawn_posix.py”, line 32, in <strong>init</strong><br>
super().<strong>init</strong>(process_obj)<br>
File “/opt/conda/lib/python3.6/multiprocessing/popen_fork.py”, line 19, in <strong>init</strong><br>
self._launch(process_obj)<br>
File “/opt/conda/lib/python3.6/multiprocessing/popen_spawn_posix.py”, line 42, in _launch<br>
prep_data = spawn.get_preparation_data(process_obj._name)<br>
File “/opt/conda/lib/python3.6/multiprocessing/spawn.py”, line 143, in get_preparation_data<br>
_check_not_importing_main()<br>
File “/opt/conda/lib/python3.6/multiprocessing/spawn.py”, line 136, in _check_not_importing_main<br>
is not going to be frozen to produce an executable.’’’)<br>
RuntimeError:<br>
An attempt has been made to start a new process before the<br>
current process has finished its bootstrapping phase.</p>
<pre><code>    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.</code></pre>

EliB: <p>I figured out:<br>
a script that goes to multiple gpu has to include main. Works now</p>
Properly define a non differentiable layer

Jonodera: <p>Hi,</p>
<p>I would like to create a layer containing non differentiable operations (such as a K-means for example) and I would like to combine it with deep neural network (pre-)trained using SGD.<br>
How to properly and clearly create using PyTorch, my layer and define it as non differentiable in order that the gradient is not be computed.</p>
<p>Thank you</p>
why the result of torch.nn.Linear() isn't consistent when the length of data is different?

hxwu: <p>for example:<br>
fc = torch.nn.Linear(128,512)<br>
data1 = torch.randn(4,512,128)<br>
data2 = data1[0:1,200,128)<span class="hashtag-raw">#the</span> length of data is changed<br>
result1 = fc(data1)[0:1,200,:]<br>
result2 = fc(data2)<br>
print(result1-result2)<span class="hashtag-raw">#the</span> result1 and result2 should be the same, however they are different in 10^-7</p>
<p>I didn’t find any reason for this result, I hope someone can help me. Thank You.(torch version is 1.1.0)</p>
<p>the result is different in the conditions shown in the following image:<img src="https://discuss.pytorch.org/uploads/default/original/2X/4/493f6df6c8d8f9e20620d2ceabbe2f3e16f75717.png" alt="image" data-base62-sha1="arYKEGDJs63TqGZP9EBd7lDbA7t" width="557" height="216"></p>

ptrblck: <p>Floating point numbers have a limited precision, which results in these small differences.<br>
If you need more precision for whatever reason, you could cast your tensors to <code>.double()</code>.<br>
Have a look at <a href="https://blog.demofox.org/2017/11/21/floating-point-precision/" rel="nofollow noopener">this nice blog post</a> about floating point precision.</p>
Softmax Dim parameter

WeiQin_Chuah: <p>Hi,</p>
<p>So say I have a 3D tensor <code>[Channels x Width x Height]</code> and I would like to perform softmax along the channel dimension. Do I specify the parameter <code>dim=0</code> or <code>dim=2</code>?</p>
<p>Thank you</p>

Nikronic: <p>Hi,</p>
<p>We read dimensionality left to right, so in your case it have to be <code>dim=0</code>.</p>
<p>Good luck</p>
Error message when loading state dict

RuFAI: <p>Hi, I want to load my saved model as described in the tutorials and it worked very well a few days ago. But now I get the following error message</p>
<pre><code class="lang-auto">optimal_network.load_state_dict(t.load('trained_Model_Dropout_Test.pt')) 

  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 751, in load_state_dict
    state_dict = state_dict.copy()

AttributeError: 'function' object has no attribute 'copy'
</code></pre>
<p>The Code where it happens is:</p>
<pre><code class="lang-auto">def trace_Model(self):
        optimal_network = Network()
        optimal_network.load_state_dict(t.load('trained_Model_Dropout_Test.pt')) #error here    
        traced_model = t.jit.trace(optimal_network, t.rand(1,1,128,128)) 
        print(f"is Training Mode: {optimal_network.training}")
        traced_model.save("traced_Model_Dropout_Test.pt")
</code></pre>
<p>It seems that there is a bug in the nn.Module class. How can I fix this? I get also the Message, that Code has been changed and I tried the advice but it did not help.<br>
<code>SourceChangeWarning: source code of class '__main__.Network' has changed. you can retrieve the original source code by accessing the object's source attribute or set `torch.nn.Module.dump_patches = True` and use the patch tool to revert the changes. warnings.warn(msg, SourceChangeWarning)</code></p>
Different workers act exactly the same

007: <p>When I implement DPPO algorithm in pytorch, I found that different workers act exactly the same, and I don’t know why. In my game environment, every worker faces the same initial state, and I use torch.distributions.sample function to sample actions in worker. In ‘main’ function, I use</p>
<p>np.random.seed(params.seed)<br>
torch.manual_seed(params.seed)<br>
torch.backends.cudnn.deterministic = True<br>
torch.backends.cudnn.benchmark = False</p>
<p>to guarantee the reproducibility of the algorithm. So I think it might have some influence on the variation between workers. And I wonder how to solve that problem. Help me!</p>

007: <p>The main、model、distributions function is as follows:</p>
<ol>
<li>main: <a href="https://paste.ubuntu.com/p/JFDgrrkWPF/" rel="nofollow noopener">https://paste.ubuntu.com/p/JFDgrrkWPF/</a>
</li>
<li>model: <a href="https://paste.ubuntu.com/p/NgBdXPgdnb/" rel="nofollow noopener">https://paste.ubuntu.com/p/NgBdXPgdnb/</a>
</li>
<li>distribution: <a href="https://paste.ubuntu.com/p/3Y9fMc6HzF/" rel="nofollow noopener">https://paste.ubuntu.com/p/3Y9fMc6HzF/</a>
</li>
</ol>

007: <p>Problem solved! Thanks to the code :<a href="https://github.com/ikostrikov/pytorch-a3c/blob/master/train.py" rel="nofollow noopener">https://github.com/ikostrikov/pytorch-a3c/blob/master/train.py</a><br>
I added     torch.manual_seed(params.seed + rank) in each of the worker to guarantee the variation between different workers and the reproduction of my algorithm.</p>
A problem about optimizer.param_groups in step function

Chinesexingyili: <p>I hope to rewrite a step function according to my own need.<br>
In this function, I need to new some tensors which are the same size to all weights’ tensors(exclude biases’ tensors).<br>
I hope to get the size from elements in param_groups.But I can’t realize this dict’s elements’ meaning.<br>
Someone can explain this?or where can I learn the information about this?</p>

jpeg729: <p>When you initialize the optimizer using</p>
<pre><code class="lang-auto">optimizer = torch.optim.SGD(model.parameters(), lr=0.1)
</code></pre>
<p>or similar, pytorch creates one param_group. The learning rate is accessible via <code>param_group['lr']</code> and the list of parameters is accessible via <code>param_group['params']</code></p>
<p>If you want different learning rates for different parameters, you can initialise the optimizer like this.</p>
<pre><code class="lang-auto">optim.SGD([
                {'params': model.base.parameters()},
                {'params': model.classifier.parameters(), 'lr': 1e-3}
            ], lr=1e-2, momentum=0.9)
</code></pre>
<p>This creates two parameter groups with different learning rates. That is the reason for having param_groups.</p>
<p>You might find reading the source for SGD to be useful. <a href="http://pytorch.org/docs/0.3.1/_modules/torch/optim/sgd.html#SGD">http://pytorch.org/docs/0.3.1/_modules/torch/optim/sgd.html#SGD</a></p>

Chinesexingyili: <p>Thanks for your reply.I still have two problems about the source code:<br>
for p in group[‘params’]:<br>
if p.grad is None:<br>
continue<br>
d_p = p.grad.data<br>
if weight_decay != 0:<br>
d_p.add_(weight_decay, p.data)<br>
if momentum != 0:<br>
param_state = self.state[p]<br>
problem1: Each p in param_groups is a data about weights’ or biases’ tensor data or grad or other parameters(kind of class Variable?)?  So p1 is about conv1’s weight ,p2 is about conv1’s bias; p3 is about conv2’s weight; p4 is about conv2’s bias?<br>
problem2:Is there only one group which include all parameters like weights,biases,weight decay and so on?``` If so, why there is a “s” in  name param_group(s)? It’s funny.</p>

jpeg729: <p>Please use the code formatting tool in future.</p>
<ol>
<li>
<p>Each p <em>is</em> one of the parameter Variables of the model. p.grad is the Variable containing the gradients for that parameter.</p>
</li>
<li>
<p>There will be several param_groups if you specify different learning rates for different parameters when you initialize the optimizer (as explained above).</p>
</li>
</ol>

Chinesexingyili: <p>Thank you very much!<br>
I will learn how to use that tool, thank you for your reply and advice!</p>

Pranaymodukuru: <p>What if I want to use different optimizers for different param groups?</p>
<p>Do I have to define two optimizers or is there any other way?</p>
Model works fine during training, but same output on Validation Set

shubhendumishra: <p><a href="https://www.kaggle.com/shubhendumishra/recognizing-faces-in-the-wild-vggface-pytorch/" class="onebox" target="_blank" rel="nofollow noopener">https://www.kaggle.com/shubhendumishra/recognizing-faces-in-the-wild-vggface-pytorch/</a></p>
<p>Here is a kernel I’m working on. Can somebody tell me why this is happening? During training model outputs are fine but during Validation only one class is the output</p>
Sampling with masked matrix

Tako: <p>Hi All,<br>
I am struggling with sampling problem.<br>
I want to sample only some values in matrix.<br>
I would like to sample in a masked matrix.<br>
For example, There is a 10x10 matrix like below.<br>
[[0.894 0.107 0.94  0.801 0.793 0.992 0.437 0.    0.    0.   ]<br>
[0.444 0.956 0.002 0 0.043 0.504 0.904 0.    0.    0.   ]<br>
[0.978 0.22  0 0.918 0.342 0.168 0.927 0.    0.    0.   ]<br>
[0.334 0.166 0.094 0.853 0.619 0.58  0.745 0.    0.    0.   ]<br>
[0.021 0  0.23  0.232 0.339 0.794 0.351 0.    0.    0.   ]<br>
[0.758 0.566 0.01  0.141 0.306 0.766 0.757 0.    0.    0.   ]<br>
[0.71  0 0 0.738 0.6   0.154 0.917 0.    0.    0.   ]<br>
[0.757 0.573 0.324 0.04  0.075 0.216 0.872 0.    0.    0.   ]<br>
[0.    0.    0.    0.    0.    0.    0.    0.    0.    0.   ]<br>
[0.    0.    0.    0.    0.    0.    0.    0.    0.    0.   ]]<br>
It is not necessarily zero.<br>
I want to sample only in non-masking parts.<br>
I want to do sample for each row. Ex) torch.multinomial<br>
Please give me some tips for this task.<br>
Thanks.</p>
Mask multiplication

sajastu: <p>Hi everybody,</p>
<p>I’m working with Seq2seq networks for NLP purposes and almost new to this Deep Learning area. Just have a question about matrix multiplication on Pytorch.</p>
<p>Suppose that I have these vectors:</p>
<p><code>a = shape(2, 34, 200)</code></p>
<pre><code class="lang-auto">b = shape (2,6, 200)
</code></pre>
<pre><code class="lang-auto">mask = shape(2,34)
</code></pre>
<p>mask is a 0-1 vector denoting specific tokens in a vector. (1: exists, 0: otherwise)<br>
Here is the work I wanna do: I want to use mask vector to find the associated rows (denoted by 1 in mask) and multiply these rows by ‘b’ vector. As for this discussion, b vector associates with those rows in the a vector that has value 1 in the mask.</p>

tom: <p>I think you need to be a bit more specific here:</p>
<ul>
<li>what is your desired output shape?</li>
<li>what should happen when mask=0? Should for those the corresponding results be 0 or …?</li>
</ul>
<p>Best regards</p>
<p>Thomas</p>
MaxPool1D shape calculation

charlesll: <p>Hi,</p>
<p>I am trying to implement a 1D CNN network for 1D signal processing. I managed to implement a simple network taking some input and giving me an output after processing in a conv1D layer followed by a fully connected relu output layer.</p>
<p>However, I wanted to apply MaxPool1d and I get in trouble with the size of its output, necessary to calculate the input size of the fully connected output layer. It seems that if stride = kernel_size, then for odd input length, my implementation of the <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.MaxPool1d" rel="nofollow noopener">formula</a> provided in the docs does not allow to calculate properly the output size of MaxPool1d.</p>
<p>The following code illustrates this:</p>
<pre><code class="lang-auto">import numpy as np
import torch

# Some fake X input of dimension (100 samples, 1 channel, 1999 length)
X = torch.FloatTensor(np.ones((100,1,1999)))

# init parameters
input_channels = 1

# First conv layer parameters
out_channels_conv1 = 10 # number of kernels
Conv1_padding = 1
Conv1_dilation = 1
Conv1_kernel_size = 3
Conv1_stride = 1

# MasPool parameters
MaxPool_kernel_size = 3
MaxPool_stride = 3
MaxPool_padding = 0
MaxPool_dilation = 1

input_size = X.shape[2]

# first conv layer
conv1 = torch.nn.Conv1d(input_channels, out_channels_conv1, Conv1_kernel_size, stride=Conv1_stride, padding = padding)
out1 = conv1(X)

# calculating the output size of Conv1 with the formula from Pytorch docs
L_out = ((input_size + 2*Conv1_padding - Conv1_dilation*(Conv1_kernel_size-1) -1)/Conv1_stride +1)
if int(L_out) == out1.shape[2]:
  print("Length at output of conv1 equals calculated length so everything looks good.\n Now pushing output of conv1 in MaxPool1d...")
  
# Now pushing data through MaxPool1d
MP1 = torch.nn.MaxPool1d(MaxPool_kernel_size,stride=MaxPool_stride)
out2 = MP1(out1)
print("Observed length is {}".format(out2.shape[2]))
Lout2 = (L_out+2*MaxPool_padding-MaxPool_dilation*(MaxPool_kernel_size-1)-1)/MaxPool_stride+1
print("Calculated length with Pytorch doc is {}".format(Lout2))
</code></pre>
<p>Output highlights that observed length is 666 while calculated length with the formula from Pytorch doc is 666.33…</p>
<p>Did I miss something? Should stride always be smaller than kernel_size? Or is there something wrong with my calculation?</p>

Joel_Wu: <p>you missing round down operation(⌊·⌋) in you calculation of Lout2</p>

charlesll: <p>Ah thanks, I did not notice the round down operation I misread it for brackets <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>
<p>Thanks!</p>
"no_grad" vs "enable_grade" vs "set_grad_enabled"

spnova12: <p>Can “set_grad_enabled” replace “no_grad” and “enable_grade”?</p>

ptrblck: <p><code>set_grad_enabled</code> can be used as a function or context manager as an alternative for <code>torch.no_grad</code> and <code>torch.enable_grad</code>.</p>

spnova12: <p>Thank you very much for your reply!</p>
How to reshape tensor shape?

skyunyoo: <p>I’m dealing with segmentation now.<br>
I want to use with cross_entropy_loss and dice_loss. The form of input shape is[4,512,512], having value of 0,1 in indices and the output form is [4,2,512,512].</p>
<blockquote>
<p>masks_shape → torch.Size([4, 512, 512])<br>
output_masks_shape → torch.Size([4, 2,512, 512])</p>
</blockquote>
<p>I have got the dice_loss function from another site.</p>
<pre><code class="lang-auto">def dice_loss(input, target):
    smooth = 1.
    loss = 0.
    for c in range(n_classes):
           iflat = input[:, c ].view(-1)
           tflat = target[:, c].view(-1)
           intersection = (iflat * tflat).sum()
           
           w = class_weights[c]
           loss += w*(1 - ((2. * intersection + smooth) /
                             (iflat.sum() + tflat.sum() + smooth)))
    return loss
</code></pre>
<p>How to reshape input shape [4,512,512] to [4,2,512,512] including indices(0,1)??</p>
<p>And I want to use class_weights because I’m dealing with an unbalanced classes.<br>
How to calculate class_weights??</p>
How to properly setup pytorch text BPTTIterator

bachir: <p>I’m trying to train a language model using pytorch text then using <code>BPTTIterator</code> to get an batch iterator for training. However, I’m getting an empty one (i.e. length 0).</p>
<pre><code class="lang-auto">spacy_en = spacy.load('en')

def tokenizer(text):
    return [tok.text for tok in spacy_en.tokenizer(text)]

# embedding dimension (size of the word vectors)
embed_size = 300
# batch size
batch_size = 64
# BPTT (backpropagation through time) length
seq_len = 50

TEXT = tt.data.Field(tokenize=tokenizer, lower=True, batch_first=True) #sequential=True

train, valid = tt.data.TabularDataset.splits(path=PATH, train='train.csv', validation='valid.csv', format='csv', fields=[('text', TEXT)])

TEXT.build_vocab(train, vectors='fasttext.en.300d')

vocab = TEXT.vocab

train_iter, valid_iter = tt.data.BPTTIterator.splits((train, valid), batch_size=batch_size, bptt_len=30)
</code></pre>
<p>Now both <code>len(train_iter)</code> and <code>len(valid_iter)</code> gives 0!</p>
<p>My dataset is something like the following, and I’m trying to prepare it to pass it thought an RNN</p>
<pre><code class="lang-auto">A child in a pink dress is climbing up a set o...
A girl going into a wooden building .
A little girl climbing into a wooden playhouse .
A little girl climbing the stairs to her playh...
A little girl in a pink dress going into a woo...
</code></pre>
<p>What I’m missing here? i’m almost following this official example which works perfectly on wikipedia datasets - <a href="https://github.com/pytorch/text/blob/master/test/language_modeling.py" rel="nofollow noopener">link</a></p>

bachir: <p>It seems that <code>BPTTIterator</code> expects a dataset of one example, so I transformed my training dataset as follows:</p>
<pre><code class="lang-auto">def dataset2example(dataset):
    examples = list(map(lambda example: ['_bos_']+ example.text + ['_eos_'], dataset.examples))
    examples = [item for example in examples for item in example]
    example = tt.data.Example()
    setattr(example, 'text', examples)
    return tt.data.Dataset([example], fields={'text': TEXT})

train_example = dataset2example(train)
valid_example = dataset2example(valid)
</code></pre>
<p>Now I can pass those <code>train_example</code> to the <code>BPTTIterator</code> like this:</p>
<pre><code class="lang-auto">train_iter, valid_iter = tt.data.BPTTIterator.splits((train_example, valid_example), batch_size=batch_size, bptt_len=30)
</code></pre>
How can I implement Pytorch in c programming ? 

an99: <p>Hi , I want to ask about How can I implement Pytorch in c programming ?</p>

smth: <p>we dont support C, but we do support C++.</p>
<p>See documentation here:</p>
<p><a href="https://pytorch.org/cppdocs/" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/cppdocs/</a></p>
<p>See our examples here:</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/pytorch/examples/tree/master/cpp" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars2.githubusercontent.com/u/21003710?s=400&amp;amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/pytorch/examples/tree/master/cpp" target="_blank" rel="nofollow noopener">pytorch/examples</a></h3>

<p>A set of examples around pytorch in Vision, Text, Reinforcement Learning, etc. - pytorch/examples</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


an99: <p>Are you have another way to do it in C-programming ?!</p>
Pytorch CuDNN error

Nishanth_Sasankan: <p>Hi All,</p>
<p>Anyone know what this error is all about?</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "train.py", line 135, in &lt;module&gt;
    loss.backward()
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/torch/tensor.py", line 107, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/torch/autograd/__init__.py", line 93, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED
</code></pre>
<p>Any and all help is very much appreciated</p>
<p>Thanks</p>

ptrblck: <p>This error might point to a faulty driver installation, a failure in initializing the CUDA context or an unsupported GPU.<br>
Was your setup working before or are you using a new environment/system?<br>
In the former case, could you disable cudnn (<code>torch.backends.cudnn.enabled=False</code>), and run the code again?<br>
Maybe some other error is hidden with this error message.</p>
Torchaudio import error

Kai_Qu: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/5/5210368c0ed0f57c4f7c105960a0fdb12e0b8a0f.png" data-download-href="https://discuss.pytorch.org/uploads/default/5210368c0ed0f57c4f7c105960a0fdb12e0b8a0f" title="31%20PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/5/5210368c0ed0f57c4f7c105960a0fdb12e0b8a0f_2_689x284.png" alt="31%20PM" data-base62-sha1="bHXT72Fhx1pKQ7y6A2zMSokG8aj" width="689" height="284" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/5/5210368c0ed0f57c4f7c105960a0fdb12e0b8a0f_2_689x284.png, https://discuss.pytorch.org/uploads/default/optimized/2X/5/5210368c0ed0f57c4f7c105960a0fdb12e0b8a0f_2_1033x426.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/5/5210368c0ed0f57c4f7c105960a0fdb12e0b8a0f_2_1378x568.png 2x" data-dominant-color="F5F6F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">31%20PM</span><span class="informations">2296×946 260 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Hi, I tried to install torchaudio by using the command “conda install -c derickl torchaudio” from <a href="https://anaconda.org/derickl/torchaudio" class="inline-onebox" rel="noopener nofollow ugc">Login :: Anaconda.org</a>, but when I tried to import torchaudio, the error message in the screenshot popped up. Can someone help to solve this? Thanks in advance!</p>

jamarshon: <p>I ran into an issue like that before. I think upgrading the pytorch version helped but can’t remember exactly</p>

tom: <p>The error message indicates a mismatch between the PyTorch version expected by torchaudio and the one installed.</p>
<p>If possible, following the instructions on the web page would likely work best: <a href="https://github.com/pytorch/audio/#dependencies" rel="nofollow noopener">https://github.com/pytorch/audio/#dependencies</a></p>
<p>Generally, compiling the auxiliary libraries (torchvision, torchaudio) is much easier and faster than it is PyTorch itself, but I don’t know about mac specifics.</p>
<p>Best regards</p>
<p>Thomas</p>

smth: <p>it looks like the package is unofficial one and they did not declare minimum and maximum pytorch version properly.</p>
<p><a class="mention" href="/u/jamarshon">@jamarshon</a> is soon uploading official torchaudio packages to conda channel</p>

jmsosullivan: <p>I have the same problem. Has this been resolved?</p>
Caffe2 / R2Plus1D: "VideoInput is not a registered operator."

kateiyas: <p>After having no luck with various manual build manuals for Caffe2 and OpenCV (within my anaconda python 2.7 environment, Ubuntu 16.04), I followed the advice to install the nightly build binaries via:</p>
<pre><code class="lang-auto">conda install pytorch-nightly torchvision -c pytorch
</code></pre>
<p>I want to use models from <a href="https://github.com/facebookresearch/R2Plus1D" rel="nofollow noopener">R2Plus1D</a> for extracting action recognition features. However, when running their <code>extract_features.py</code> it will throw the following error message:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "lib/extract_features.py", line 333, in &lt;module&gt;
    main()
  File "lib/extract_features.py", line 328, in main
    ExtractFeatures(args)
  File "lib/extract_features.py", line 131, in ExtractFeatures
    devices=gpus,
  File "/anaconda2/lib/python2.7/site-packages/caffe2/python/data_parallel_model.py", line 34, in Parallelize_GPU
    Parallelize(*args, **kwargs)
  File "/anaconda2/lib/python2.7/site-packages/caffe2/python/data_parallel_model.py", line 219, in Parallelize
    input_builder_fun(model_helper_obj)
  File "lib/extract_features.py", line 106, in input_fn
    use_local_file=args.use_local_file,
  File "/models/c3d/R2Plus1D/lib/utils/model_helper.py", line 120, in AddVideoInput
    data, label, video_id = model.net.VideoInput(
  File "/anaconda2/lib/python2.7/site-packages/caffe2/python/core.py", line 2171, in __getattr__
    ",".join(workspace.C.nearby_opnames(op_type)) + ']'
AttributeError: Method VideoInput is not a registered operator. Did you mean: []
</code></pre>
<p>Some issues mention that there is a simple solution. Does somebody know a solution?</p>

oLIVIa-Ld: <p>i got the same problem, have you solve this?<br>
please help me,</p>
<p><a href="mailto:oliviald@126.com">oliviald@126.com</a></p>
Difference between using random sampler at Dataset vs DataLoader

prabhuteja12: <p>Hi,<br>
I’m trying to understand the difference between a <code>Dataset</code> and <code>DataLoader</code> for a specific case. The <code>DataLoader</code> allows us to specify a sampler.</p>
<p>Lets say I’ve a sampler that looks like the following.</p>
<pre><code class="lang-auto">class MyAwesomeSampler(Sampler):
    def __init__(self, indices):
        self.indices = indices
    
    def __iter__(self):
        return iter(self.indices)

    def __len__(self):
        return len(self.indices)
</code></pre>
<p>and then I give plug an instance of <code>MyAwesomeSampler</code> to the input of <code>DataLoader</code></p>
<p>How is this different from using <code>Subset(mydataset, indices)</code> where the indices are the same as above.<br>
Does this make any difference in <code>Dataset</code> objects with data-augmentation (eg any of the image datasets in torchvision).</p>

ptrblck: <p>Your custom sampler and the <code>Subset</code> will yield the same data samles, if you don’t use <code>shuffle=True</code> for the <code>DataLoader</code> using the <code>Subset</code>.</p>
<p>The main difference is, that a custom sampler can do much more than just returning a subset of the data.<br>
E.g. the <code>WeightedRandomSampler</code> implements a sample weight, which can be used to balance the batches for an imbalanced dataset.</p>
<p>It shouldn’t make any difference regarding data augmentation, as only the indices passed to <code>__getitem__</code> will be customized, not the method itself.</p>

prabhuteja12: <p>Thank you for clarifying!</p>
Correct way to save model while training

adishegde: <p>I wanted to save my model while training every few epochs and was wondering about the best way to go about it. The approach suggested in <a href="https://discuss.pytorch.org/t/how-to-save-a-model-from-a-previous-epoch/20252/3">this link</a> seems to be a common/popular way to do so.</p>
<p>However, I don’t fully understand how the above method works. By calling <code>model.cpu</code> and then <code>model.cuda</code> won’t we be creating new objects for the parameters different from the ones before calling either of the two functions as suggested in the <a href="https://pytorch.org/docs/stable/nn.html?highlight=cpu#torch.nn.Module.cuda" rel="nofollow noopener">docs</a>? If this is true, then won’t we need to change the parameters the optimizer updates? I’ve not seen this being mentioned anywhere though.</p>
<p>I’m just getting started with PyTorch and so I apologize for any ignorance on my part.</p>

ptrblck: <p>The push to the CPU and back to GPU shouldn’t be a problem, as the <code>id</code> of the parameters shouldn’t change, thus the optimizer still holds valid references to the parameters.<br>
However, I would suggest to use the same device after storing the <code>state_dict</code>, since internal states of the optimizer (e.g. using Adam) will also be stored on the initial device.</p>

adishegde: <p>Thanks! Yeah, I’ll make sure to use the same device.</p>
<p>Just a quick follow up though, what do the <code>id</code> of the parameters depend on?</p>
<pre><code class="lang-auto">model_1.cpu()
model_2.cpu()

model_2.gpu()
model_1.gpu()
</code></pre>
<p>Would the <code>id</code> of the parameters change in this case? Just trying to avoid possible mistakes.</p>
Different between torch.tensor and torch.tensor.data?

andy_li: <p>while running this code</p>
<pre><code>for e in range(10): 
  y_ = linear_model(x_train)
  loss = get_loss(y_, y_train)
  w.grad.zero_() 
  b.grad.zero_() 
  loss.backward()
  
  w = w - 1e-2 * w.grad 
  b = b - 1e-2 * b.grad 

  print('epoch: {}, loss: {}'.format(e, loss.item()))
</code></pre>
<p>my notebook raise an error that the w and b’s grad is None</p>
<p>and in this code</p>
<pre><code>w.data = w.data - 1e-2 * w.grad.data 
b.data = b.data - 1e-2 * b.grad.data
</code></pre>
<p>the grad didn’t disappear.<br>
why former one doesn’t work?Thanks for anyone pointing out the reason.</p>

ptrblck: <p>The error is raised, since the <code>.grad</code> attribute is set to <code>None</code> be default. The first <code>backward()</code> call populates the <code>.grad</code> attribute of all parameters used in the computation of the loss, which require gradients.<br>
If you gut <code>w.grad.zero_()</code> after the <code>loss.backward()</code> call, the error should disappear.</p>
<p>Also, have a look at <a href="https://pytorch.org/tutorials/beginner/pytorch_with_examples.html#pytorch-tensors-and-autograd" rel="nofollow noopener">this example</a> to see how to manually update the parameters. E.g. in your example the <code>torch.no_grad()</code> guard is missing.</p>
Variable length of text with cnn

Lior: <p>Hello,<br>
I am a newbie with pytorch and I am wondering what is the  best practice for variable length sentences sequences in CNNs.</p>
<p>I want to use CNN for feature selection on top of word emmbeding layer and then feed the output into LSTM .</p>
<p>Now I know pytorch has a dynamic graph and I was wondering if there is a way to this except padding .</p>
why there's SparseFtrlOp and FtrlOp but only GFtrlOp without SparseGFtrlOp

Qiqin_Zhan: <p><a href="https://github.com/pytorch/pytorch/blob/519570def878eb48cc9a50183f90f7009c3afdb8/caffe2/sgd/ftrl_op.cc#L60" rel="nofollow noopener">FtrlOp</a></p>
<p><a href="https://github.com/pytorch/pytorch/blob/519570def878eb48cc9a50183f90f7009c3afdb8/caffe2/sgd/ftrl_op.cc#L84" rel="nofollow noopener">SparseFtrlOp</a></p>
<p><a href="https://github.com/pytorch/pytorch/blob/519570def878eb48cc9a50183f90f7009c3afdb8/caffe2/sgd/gftrl_op.cc#L72" rel="nofollow noopener">GFtrlOp</a></p>
Avoid parameter sharing in for-loop

sudocoder: <p>Does the following cause parameter sharing between the conv layers?</p>
<pre><code class="lang-auto">self.conv = nn.Conv2d(channels_in, channels_out)

layers = []
for _ in range(num_iters):
    layers.append(self.conv)

self.model = nn.Sequential(*layers)
</code></pre>
<p>If so, does the following remedy it?</p>
<pre><code class="lang-auto">layers = []
for _ in range(num_iters):
    layers.append(nn.Conv2d(channels_in, channels_out))

self.model = nn.Sequential(*layers)
</code></pre>

ptrblck: <p>In the first code snippet you will use the same conv layer <code>num_iters</code> times, so yes the parameters will be shared (or rather reused).<br>
The second code snippet created <code>num_iters</code> new conv layers, each containing new parameters.</p>

sudocoder: <p>Great, thank you so much!</p>
One batch many backpropagation

slavavs: <p>if i have batch = 100, can i make backpropagation for each sample of 100 individually? That is, the forward pass once, backpropagation 100 items?</p>

JuanFMontesinos: <p>You can. Pytorch typically computes element-wise loss, then average and then backprop.</p>
<p>At the time of defining the loss just choose the proper option not to compute the average and loss will return 100 elements instead of a single number.</p>
<p>Then just backprop each element like element.backward()<br>
You will have to set retain_graph=True .</p>
<p>for element in loss:<br>
element.backward(retain_graph=True)</p>

slavavs: <p>if i use MSELoss, which is the correct option?</p>
<p>“none” does not work with the batch.<br>
Leave only the “sum” and “mean”, but they will not give the desired result. Most likely I will have to write my loss function.</p>

Sunshine352: <p>It’s not necessary to use for-loop,<br>
mse_criterion = <code>torch.nn.</code> <code>MSELoss</code> (<em>reduction=‘none’</em> )<br>
loss = mse_criterion(input, label) # input-&gt;BxCxHxW, loss-&gt;Bx1 or B,<br>
loss.backward(gradient=torch.ones_like(loss))<br>
Then, loss.grad will be individual grad for each sample.</p>
How to define a deconvolution layer?

EsamOthman: <p>Hii</p>
<p>I’m new in pytorch, any help will be appreciated.<br>
I want to use deconvolution to estimate an image for 1-D feature. The same as DWT (discrete wavelet transform).</p>
<p>Thnx</p>

Jing: <p><code>torch.nn.ConvTranspose2d</code>  can do unsampling and can be  regarded as a deconvolution operation.</p>

EsamOthman: <p>Thnx for reply</p>
<p>I tried modifying this code but it gives me error.</p>
<pre><code>    self.layer1 = nn.Sequential(
        nn.Conv2d(1, 16, kernel_size=5, padding=2),
        nn.BatchNorm2d(16),
        nn.ReLU(),
        nn.MaxPool2d(2))
</code></pre>
<p>Kindly, can u modify this code so it can be used for deconvolution?</p>

Brando_Miranda: <p>do you have a code example?</p>
Drastic speed differences between 2 servers

Puumanamana: <p>I am running PyTorch on two different machines (no GPU), and I observe drastic speed differences between the two. Both machines are running Ubuntu 18.04, Python 3.7.3 with Anaconda and Pytorch v1.2.0 was built from source.</p>
<p>Here is a small sample of code to reproduce my issues:</p>
<pre><code class="lang-auto">import torch
from time import time

torch.set_num_threads(10)

N_samples, N_features = (128,5000)

x = [ torch.randn(N_samples, N_features) for _ in range(10) ]
y = [ torch.randn(N_samples) for _ in range(10) ]

model = torch.nn.Sequential(
    torch.nn.Linear(N_features, 500),
    torch.nn.ReLU(),
    torch.nn.Linear(500, 100),
    torch.nn.ReLU(),
    torch.nn.Linear(100, 1),
    torch.nn.ReLU()
)

def run_batch(xi,yi):
    t0 = time()
    model(xi)
    print(time()-t0)

[ run_batch(xi,yi) for xi,yi in zip(x,y) ]
</code></pre>
<p>On the first machine, I get:</p>
<pre><code class="lang-auto">0.45372891426086426
0.37187933921813965
0.37082910537719727
...
</code></pre>
<p>On the second, I get:</p>
<pre><code class="lang-auto">0.013288736343383789
0.009611368179321289
0.009567499160766602
...
</code></pre>
<p>For the slower machine, Python was compiled with <code>--enable-optimizations</code>, but I’m not sure for the faster one.</p>
<p>Moreover, the slower machine has more CPUs and more RAM memory than the other one.</p>
<p>Any idea where the difference might come from?</p>
<p>Thank you for your help</p>
RuntimeError: Expected object of scalar type Double but got scalar type Float for sequence element 1 in sequence argument at position #1 'tensors'

KUSHAL_JAIN: <p>I am facing this error wherein my code runs for sometime, i.e I get running loss values for some batches but midway, again this runtime error pops up. I have converted my inputs to float as mentioned in many other similar posts. Kindly help me out.</p>
<pre><code class="lang-auto">epochs = 1
for epoch in range(epochs):
    running_loss = 0
    #init_hidden_states = model.init_hidden(4)
    #dtype = torch.FloatTensor
    for embeddings, labels in dataloader:
        embeddings[0],embeddings[1],labels = embeddings[0].float(),embeddings[1].float(),labels.float()
       
        #embeddings[0] = embeddings[0].float()
        #embeddings[1] = embeddings[1].float()
        print(embeddings[0].dtype)
        #embeddings[0],embeddings[1], labels = embeddings[0].cuda(), embeddings[1].cuda(),labels.cuda()
        labels = torch.argmax(labels,dim=1)
        optimizer.zero_grad()
        #h = tuple([each.data for each in init_hidden_states])
        logits = model.forward(embeddings)
        loss = criterion(logits,labels)
        loss.backward()
        optimizer.step()
        
        running_loss += loss.item()*embeddings[0].size(0)
        #print(running_loss)
    else:
        print(f"Training loss: {running_loss}")
</code></pre>
<pre><code class="lang-auto">&lt;ipython-input-111-2792e98997e4&gt; in &lt;module&gt;()
      4     #init_hidden_states = model.init_hidden(4)
      5     #dtype = torch.FloatTensor
----&gt; 6     for embeddings, labels in dataloader:
      7         embeddings[0],embeddings[1],labels = embeddings[0].float(),embeddings[1].float(),labels.float()
      8
</code></pre>

ptrblck: <p>Could you post some information about your <code>Dataset</code>?<br>
E.g. what kind of data are you using and how did you create it?</p>
Getting access to which device is computing which layer after wrapping model with DataParallel?

sree_harsha: <p>Hi!</p>
<p>So this is an updated question:</p>
<p>How can I get access to which device and what gradient it is computing when using DataParallel?<br>
How would I be able to print</p>
<p>"Device 1: "[ …]<br>
"Device 2: " […]<br>
For the same layer?</p>
<p>Thanks.</p>

sree_harsha: <p>Bump. Updated question.</p>
<p>Would appreciate any help.</p>

ptrblck: <p>I’m not sure I’m understanding the question completely, but if you would like to print the current input and parameter device, you could add:</p>
<pre><code class="lang-python">def forward(self, x):
    print(x.device, self.fc.weight.device, ...)
</code></pre>
<p>to your <code>forward</code> method.</p>
Transforming dataset not working?

ajhanwar: <p>Hey all! I’m using the MNIST dataset available through <code>torchvision</code> and trying to use transform operations to create synthetic data.</p>
<p>In addition to a regular <code>train_set</code> where I only used <code>transforms.ToTensor()</code>, I wrote the following with the intention of appending it to the original <code>train_set</code>:</p>
<pre><code class="lang-auto">train_set2 = torchvision.datasets.MNIST(
    root='./data',
    train=True,
    download=True,
    transform=transforms.Compose([
        transforms.RandomAffine(degrees=20, 
                                translate=(0.9, 0.9), 
                                scale=(0.9, 1.1), 
                                shear=(-20, 20)),
        transforms.ToTensor()
    ])
)
</code></pre>
<p>However, when  I view the images that are produced through the extracting and transforming of the dataset there does not appear to be any difference in how the images look at all.</p>
<p>For example, the following are my results:</p>
<p><code>plt.imshow(train_set.data[0])</code><br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/0/0d54932882708188c22b6eca24cf535720c71793.png" alt="image" data-base62-sha1="1TVphQoVc975oYbrDDvzr1P4mxd" width="255" height="252"></p>
<p><code>plt.imshow(train_set2.data[0])</code><br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/0/0d54932882708188c22b6eca24cf535720c71793.png" alt="image" data-base62-sha1="1TVphQoVc975oYbrDDvzr1P4mxd" width="255" height="252"></p>
<p>Any clarification would be greatly appreciated!</p>

John_Deterious: <p>How could you send torch.tensor data type to plt.imshow? something smells here</p>

ajhanwar: <p>Well when I convert it to a numpy array it produces the same results… more so confused about why there isn’t any sort of transformation of the data occurring</p>

ptrblck: <p>You are directly indexing the underlying non-transformed data using <code>train_set2.data</code>.<br>
Try to index the <code>Dataset</code>, to get the transformed tensors <code>train_set2[index]</code>.</p>
How to 4D tensor dot product?

wonchulSon: <p><code>torch.dot(nn.Conv2d(3, 64, kernel_size=(3,1), stride=1, padding=(0,1)).weight, nn.Conv2d(3, 64, kernel_size=(1,3), stride=1, padding=(0,1)).weight)</code></p>
<p><code>RuntimeError: dot: Expected 1-D argument self, but got 4-D</code></p>
<p>Like this, I want to make 3x3 filter and apply it to train.<br>
How should I do?</p>
<p>Thank you.</p>

ptrblck: <p>The dot product will return a scalar value, so maybe you would like to apply <code>torch.matmul</code> on the filter kernels?</p>
1D and 2D Histograms in PyTorch

Roulbac: <p>Hi,</p>
<p>I am trying to implement an efficient histogram method in PyTorch.<br>
I know PyTorch already has a histc and bincount though there is no 2D version of that.</p>
<p>I am looking to implement it in parallel to be fast; i.e for a vector of k values and using n bins, I want to do k*n operations in parallel.</p>
<p>I have looked at max grid &amp; block size for CUDA and it is doable giving my number of bins and size of vector.</p>
<p>How should I go about and do this to be integrated with PyTorch?</p>
How to do the weight sharing between two classes?

nkcdy: <p>In my architecture, i have to reuse a module with the same weights as its original ones. Does the following method is the right way to realize my thought?</p>
<p>Class Encoder(nn.module)：<br>
def <strong>init</strong>(self):<br>
super(Encoder, self).<strong>init</strong>()<br>
。。。<br>
def forward(self, x1, x2, emb, input_sel)：<br>
if input_sel == 0 ：<br>
x = x1<br>
else：<br>
x = x2</p>
<pre><code>       x = 。。。。
       。。。
</code></pre>
<p>Class Top（nn.module）：<br>
def <strong>init</strong>(self):<br>
super(Top, self).<strong>init</strong>()<br>
self.encoder = Encoder()<br>
self.decoder = Decoder()<br>
。。。<br>
def forward(input, emb1,emb2):<br>
encoder_output = self.encoder(input,input,emb1,0)<br>
decoder_output = self.decoder(encoder_output)<br>
code_output = self.encoder(decoder_output,decoder_output,emb1,1)<br>
return encoder_output, decoder_output, code_output</p>
<p>Can the two encoders share the same weight in above method??</p>

Mazhar_Shaikh: <p>Hi nkcdy,<br>
Yes, the code that you’ve shared would be one way to have shared weights, and should work for your purpose.</p>