DataParallel can not split data to different gpus

MrTuo: <p>My pytorch model can run on a single gpu correctly. But when I  use DataParallel to run it on multiple gpus ,I got an error : arguments are located on different GPUs<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/a/a3b588b886a0afbaa93a068befa9fadbb1df2ee6.png" data-download-href="https://discuss.pytorch.org/uploads/default/a3b588b886a0afbaa93a068befa9fadbb1df2ee6" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/2X/a/a3b588b886a0afbaa93a068befa9fadbb1df2ee6.png" alt="image" data-base62-sha1="nmeLhwUu2CIZZYXP3qSb0wIIVlc" width="690" height="301" data-dominant-color="121213"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1043×455 17 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
It seems that embedding parameters and input tensor are on different gpu. So I print the divice of them:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/c/c7067a5cdbde4fe35cb8ffa6ee860e3ea333a19b.png" alt="image" data-base62-sha1="soETwMOlneXdcZRvWX3Zb1hfzCz" width="261" height="98"><br>
The batch size is 12 and gpu num is 2. As is shown above, it has two problem:</p>
<ol>
<li>The batch size should be 6 instead of 12 (cause it should be split into two pices by DataParallel).</li>
<li>The embedding weigh are parallel to two gpus(cuda:0 and cuda:1) but the input data on the same gpu.</li>
</ol>
<p>I use torchtext to load data and the input data is on the cuda:0.<br>
I use the following code to parallelize model:</p>
<pre><code class="lang-auto">    self.device, device_ids = self._prepare_device(config['n_gpu'])
    self.model = model.to(self.device)
    # data parrallel
    if len(device_ids) &gt; 1:
        self.model = torch.nn.DataParallel(model, device_ids=device_ids)
...
    def _prepare_device(self, n_gpu_use):
        """
        setup GPU device if available, move model into configured device
        """
        n_gpu = torch.cuda.device_count()
        if n_gpu_use &gt; 0 and n_gpu == 0:
            self.logger.warning(
                "Warning: There\'s no GPU available on this machine, training will be performed on CPU.")
            n_gpu_use = 0
        if n_gpu_use &gt; n_gpu:
            msg = "Warning: The number of GPU\'s configured to use is {}, but only {} are available on this machine.".format(
                n_gpu_use, n_gpu)
            self.logger.warning(msg)
            n_gpu_use = n_gpu
        device = torch.device('cuda:0' if n_gpu_use &gt; 0 else 'cpu')
        list_ids = list(range(n_gpu_use))
        return device, list_ids
</code></pre>
<p>And I use the completely same logic to parallelize another model and it works!</p>

ptrblck: <p>Could you post some code of your training loop?<br>
Does your data contain another dummy dimension in <code>dim0</code>?</p>

MrTuo: <p>Thanks for reply. I have fix this problem by changing the model input. Previously, I directly put torchtext iterator object into the model:</p>
<pre><code class="lang-auto">for batch_idx, batch in enumerate(self.data_loader.train_iter):
    output = self.model(batch)
</code></pre>
<p>Then I modified the code and it works：</p>
<pre><code class="lang-auto">for batch_idx, batch in enumerate(self.data_loader.train_iter):
    input_data = {
            'q_word': batch.q_word[0],
            'q_lens': batch.q_word[1],
            'paras_word': batch.paras_word[0],
            'paras_num': batch.paras_word[1],
            'paras_lens': batch.paras_word[2],
        }
    output = self.model(input_data)
</code></pre>
<p>Or like this:</p>
<pre><code class="lang-auto">for batch_idx, batch in enumerate(self.data_loader.train_iter):
    output = self.model(batch.q_word[0], batch.q_word[1], \
                        batch.paras_word[0], batch.paras_word[1], batch.paras_word[2])
</code></pre>
<p>All the value of the <code>input_data</code> are tensor.<br>
I think the key problem is that you should feed tensor or a dic of tensors so that the DataParallel model can find the tensor and split the dimension in  <code>dim0</code>.<br>
I think that DataParallel should at least gives a warninig when the DataParallel can’t find any tensor that can be split in <code>dim0</code></p>