`Exception: process 0 terminated with exit code 1` error when using `torch.multiprocessing.spawn` to parallelize over multiple GPUs

Leockl: <p>I have the following code below using <code>torch.multiprocessing.spawn</code> to parallelize over multiple GPUs:</p>
<pre><code class="lang-auto">import numpy as np
import torch
from torch.multiprocessing import Pool, set_start_method, spawn

X = np.array([[1, 3, 2, 3], [2, 3, 5, 6], [1, 2, 3, 4]])
X = torch.DoubleTensor(X)

def X_power_func(j):
    X_power = X.cuda()**j
    return X_power

if __name__ == '__main__':
    results = spawn(X_power_func, range(4), nprocs=1)

results
</code></pre>
<p>But I am getting this error below when I run the code:</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
&lt;ipython-input-9-97eb990d7396&gt; in &lt;module&gt;()
     12 
     13 if __name__ == '__main__':
---&gt; 14     results = spawn(X_power_func, range(4), nprocs=1)
     15 
     16 results

2 frames
/usr/local/lib/python3.6/dist-packages/torch/multiprocessing/spawn.py in join(self, timeout)
    111                 raise Exception(
    112                     "process %d terminated with exit code %d" %
--&gt; 113                     (error_index, exitcode)
    114                 )
    115 

Exception: process 0 terminated with exit code 1
</code></pre>
<p>What I have done wrong in my code?</p>

iffiX: <p>First, please read API document <strong>carefully</strong>:</p>
<blockquote>
<p>torch.multiprocessing.spawn(fn, args=(), nprocs=1, join=True, daemon=False, start_method=‘spawn’)</p>
</blockquote>
<blockquote>
<p>Parameters</p>
<ol>
<li>fn (function) – …The function is called as fn(i, *args), where i is the process index and args is the passed through tuple of arguments.</li>
<li>args (tuple) – Arguments passed to fn.</li>
<li>nprocs (int) – Number of processes to spawn.</li>
</ol>
</blockquote>
<blockquote>
<p>Returns<br>
None if join is True, ProcessContext if join is False</p>
</blockquote>
<p>First, you should call your function as:</p>
<pre><code class="lang-auto">if __name__ == '__main__':
    # start with 4 processes
    # your original method will invoke your function as X_power_func(0, 0, 1, 2, 3)
    spawn(X_power_func, nprocs=4)
</code></pre>
<p>Secondly, do not put result at:</p>
<pre><code class="lang-auto">if __name__ == '__main__':
    results = spawn(X_power_func, range(4), nprocs=1)

results
</code></pre>
<p>Because in your subprocesses, they will try to access “results”, but since they are not “<strong>main</strong>”, “results” is not defined,</p>
<p>Thirdly, spawn will not return results.</p>
<blockquote>
<p>Returns<br>
None if join is True, ProcessContext if join is False</p>
</blockquote>
<p>In summary, please <strong>re-read documents</strong>, they take <strong>a lot of time to write</strong>.</p>

Leockl: <p>Many thanks <a class="mention" href="/u/iffix">@iffiX</a> for input on this. I am new to <code>torch.multiprocessing.spawn</code> and PyTorch in general, so I guess I just got confused when I read the documentation on it.</p>
<ol>
<li>With your first point above, are you missing <code>args=()</code>?, ie.</li>
</ol>
<pre><code class="lang-auto">if __name__ == '__main__':
    # start with 4 processes
    # your original method will invoke your function as X_power_func(0, 0, 1, 2, 3)
    spawn(X_power_func, args=(0, 1, 2, 3), nprocs=4)
</code></pre>
<p>I don’t understand why must we start with 4 processes?</p>
<ol start="2">
<li>
<p>With your second point, if the <code>results</code> are kept within <code>main</code>, how would I then output the <code>results</code> outside of <code>main</code>?</p>
</li>
<li>
<p>With your third point, can I ask what is <code>ProcessContext</code>? If I want to return <code>results</code>, does that mean I need to set <code>join=False</code>?</p>
</li>
</ol>
<p>I am sorry for having so many questions. Would really be grateful if you could help.</p>

iffiX: <p>Well, being new is not an excuse! <img src="https://discuss.pytorch.org/images/emoji/apple/rofl.png?v=9" title=":rofl:" class="emoji" alt=":rofl:"></p>
<ol>
<li>Yes, I am missing <code>args</code>, because torch.multiprocessing will invoke your function <code>X_power_func(rank)</code>, the default argument is the rank of the started process.</li>
<li>If you want to print results outside of main, you main print it in the invoked function:<pre><code class="lang-auto">def X_power_func(j):
     X_power = X.cuda()**j
     print(X_power)
     return X_power
</code></pre>
</li>
<li>No, in order to properly return results, you should either use <code>torch.multiprocessing.pool</code> or passing a pipe object or anything that can be used to perform inter-process communication.<br>
Torch multiprocessing module is a very thin wrapper of the original multiprocessing module, it basically just registers some customized serializers.</li>
</ol>

iffiX: <p>Or you could post your detailed requirements so that we can workout a proper solution for you.</p>

Leockl: <p>Yeah you’re right <a class="mention" href="/u/iffix">@iffiX</a>, I should mention here what I actually want to do.</p>
<p>I am trying to figure out a way to parallelize over multiple GPUs on non-neural net computations in PyTorch. More specifically, I have developed an estimator (non-neural net) in Scikit-learn and the speed performance is slow when a certain hyperparameter is increased. To solve this, I am re-writing the estimator in PyTorch so that I can make use of GPU processing and hopefully multiple GPUs as well.</p>
<p>I have posted a question here in PyTorch Forums (1-2 days ago), Stackoverflow, PyTorch Github and multiple channels in Reddit and I haven’t gotten a reply yet or no one seems to know the full solution. In fact, what I got is the opposite where a lot of folks over at Reddit wanted to know how this is done too. So far, I feel like you’re the only one who seems to know <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"></p>
<p>Anyhow, I did initially tried using <code>torch.multiprocessing.pool</code>. The MRE code is as below:</p>
<pre><code class="lang-auto">import numpy as np
import torch
from torch.multiprocessing import Pool, set_start_method

X = np.array([[1, 3, 2, 3], [2, 3, 5, 6], [1, 2, 3, 4]])
X = torch.DoubleTensor(X)

def X_power_func(j):
    X_power = X.cuda()**j
    return X_power

if __name__ == '__main__':
  set_start_method('spawn', force=True)
  with Pool(processes = 2) as p:   # Parallelizing over 2 GPUs
    results = p.map(X_power_func, range(4))
</code></pre>
<p>However when I run this code, it hangs or keeps running forever without any errors.</p>
<p>When I removed <code>set_start_method('spawn', force=True)</code>, the code ran properly and gave me the results, but this only works for when I run the code once. When I ran the code again in subsequent runs, I get the error <code>RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method</code>.</p>
<p>Someone in Reddit suggested that I should use <code>torch.multiprocessing.spawn</code> which lead me here back to PyTorch Forum with this post.</p>

iffiX: <p>This error is here because:</p>
<pre><code class="lang-auto">X = torch.DoubleTensor(X)

def X_power_func(j):
    X_power = X.cuda()**j
    return X_power
</code></pre>
<p>You are referencing a global variable here, since “fork” will map the memory of forker to forkee, you can<br>
pass this function to subprocesses correctly, however, “fork” is not compatible with cuda, and therefore the error is thrown.</p>
<p>Could you please show the full code of your estimator? Writing parallel programs in python could be a real pain, with your full code we can choose the most efficient and simplest solution.</p>

Leockl: <p>Ok, the source code for the estimator is here:<br>
</p><aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/leockl/helstrom-quantum-centroid-classifier/blob/master/hqc/hqc.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/leockl/helstrom-quantum-centroid-classifier/blob/master/hqc/hqc.py" target="_blank" rel="nofollow noopener">leockl/helstrom-quantum-centroid-classifier/blob/master/hqc/hqc.py</a></h4>
<pre><code class="lang-py">import numpy as np
from sklearn.base import BaseEstimator, ClassifierMixin
from sklearn.utils.validation import check_X_y, check_array, check_is_fitted
from sklearn.utils.multiclass import check_classification_targets
from sklearn.preprocessing import normalize
from joblib import Parallel, delayed

class HQC(BaseEstimator, ClassifierMixin):
    """The Helstrom Quantum Centroid (HQC) classifier is a quantum-inspired supervised 
    classification approach for data with binary classes (ie. data with 2 classes only).
                         
    Parameters
    ----------
    rescale : int or float, default = 1
        The dataset rescaling factor. A parameter used for rescaling the dataset. 
    encoding : str, default = 'amplit'
        The encoding method used to encode vectors into quantum densities. Possible values:
        'amplit', 'stereo'. 'amplit' means using the amplitude encoding method. 'stereo' means 
        using the inverse of the standard stereographic projection encoding method. Default set 
        to 'amplit'.
</code></pre>

  This file has been truncated. <a href="https://github.com/leockl/helstrom-quantum-centroid-classifier/blob/master/hqc/hqc.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>It has already been parallelized for multiple CPUs but it’s still slow when the hyperparameter <code>n_copies</code> is increased. This hyperparameter controls the number of times a Kronecker tensor product is performed, which will result in multiplication of very large matrices when it is increased, therefore slowing down the speed performance when this hyperparameter is large.</p>

mrshenli: <p>You can use <a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor.share_memory_" rel="nofollow noopener"><code>share_momery_()</code></a> and <code>torch.multiprocessing.SimpleQueue</code> to implement IPC. E.g.:</p>
<pre><code class="lang-python">import numpy as np
import torch
import torch.multiprocessing as mp


def func(rank, x, p2c, c2p):
    x_power = x.to(rank) ** rank
    c2p.put(x_power)
    # citing multiprocessing doc: Unlike CPU tensors, the 
    # sending process is required to keep the original tensor 
    # as long as the receiving process retains a copy of 
    # the tensor. The refcounting is implemented under the 
    # hood but requires users to follow the next best practices.
    p2c.get()
    print(f"child-{rank} done")

if __name__ == '__main__':
    nprocs = 2
    x = torch.ones(2, 2)
    x.share_memory_()
    ctx = mp.get_context('spawn')
    c2p, p2c = ctx.SimpleQueue(), ctx.SimpleQueue()
    ps = [ctx.Process(target=func, args=(rank, x, p2c, c2p)) for rank in range(nprocs)]
    [p.start() for p in ps]
    tensors = [c2p.get() for _ in range(nprocs)]
    print(tensors)
    del tensors
    for p in ps:
        p2c.put(0)
        p.join()
    print("parent done")
</code></pre>

iffiX: <p>I have read your code:</p>
<ol>
<li>So the first outerloop(function) is:<pre><code class="lang-auto">L148:             def X_prime_class_split_func(j):
</code></pre>
What’s the shape and data type of <code>X_prime_class_split[j]</code>? Maybe we could represent it as a tensor</li>
<li>from L162-L166:<pre><code class="lang-auto"> for k in range(m_class_split):
       # Encode vectors into quantum densities
       X_prime_class_split_each_row = X_prime_class_split_jth[k, :]
       density_each_row = np.dot(X_prime_class_split_each_row.reshape(-1, 1),
                                                   X_prime_class_split_each_row.reshape(1, -1))
</code></pre>
You could definetly vectorize this inner loop.</li>
<li>from L171-L174<pre><code class="lang-auto">else:
      density_each_row_copy = density_each_row
      for u in range(self.n_copies - 1):
            density_each_row = np.kron(density_each_row, density_each_row_copy)
</code></pre>
There is no efficient way to parallelize kronecker product over <code>n_copies</code> since these code are iterative and strongly serial. But you could use the <code>einsum</code> function of pytorch to calculate:<pre><code class="lang-auto">def kronecker(A, B):
    return torch.einsum("ab,cd-&gt;acbd", A, B).view(A.size(0)*B.size(0),  A.size(1)*B.size(1))
</code></pre>
</li>
<li>The scale(computation intensity) of your code does not suit process based parallelism, thread based parallelism is not good in your case as well, I suggest you "vectorize’ your code, use tensor operations insead of for loops, it would be at least 10 times more efficient.</li>
</ol>

Leockl: <p>Hi <a class="mention" href="/u/iffix">@iffiX</a>,</p>
<ol>
<li>
</li></ol>
<p><em>What’s the shape and data type of  <code>X_prime_class_split[j]</code> ?</em><br>
<code>X_prime_class_split[j]</code> is a 2d numpy array. How I have CPU parallelize my code is two-fold. First, it parallelizes over the 2 binary classes. Secondly, for each binary class, the dataset is split into batches and parallelization is performed over the batches. <code>X_prime_class_split[j]</code> is just a dataset batch, therefore it is a 2d numpy array.</p>
<p><em>Maybe we could represent it as a tensor.</em><br>
I have actually already converted all of the numpy functions in my code into PyTorch functions. It is <a href="https://github.com/leockl/HQC/blob/master/HQC%20-%20PyTorch/HQC%20-%20PyTorch.ipynb" rel="nofollow noopener">here</a> in my Github. Of course, this code is not fully working properly yet because of the issue with <code>torch.multiprocessing.Pool</code>. So in this code, <code>X_prime_class_split[j]</code> is a PyTorch tensor.</p>
<ol start="2">
<li>
</li></ol>
<p><em>You could definetly vectorize this inner loop.</em><br>
Thanks for the tip! I didn’t realize chunk of codes could also be vectorized (using <code>numpy.vectorize</code>). I googled around but couldn’t find the PyTorch equivalent of <code>numpy.vectorize</code>. If I were to vectorize this part of the code, do you know how I could do it using PyTorch tensors?</p>
<ol start="3">
<li>
</li></ol>
<p><em>There is no efficient way to parallelize kronecker product over  <code>n_copies</code>  since these code are iterative and strongly serial.</em><br>
Agree, there is no way to efficient parallelize kronecker product because of it’s iterative and serial nature. As mentioned above, my code is actually parallelized over the 2 binary classes and batches of the dataset, so I am not looking to parallelize the kronecker product.</p>
<p><em>But you could use the  <code>einsum</code>  function of pytorch to calculate…</em><br>
I have actually already done this in my Github link above! This gives me some comfort knowing that I am on the same page as you <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"></p>
<ol start="4">
<li>
</li></ol>
<p>I guess if I can’t parallelize my code over multiple GPUs, plan B would be to rewrite my code to just use 1 GPU processing.</p>
<p>Many many thanks again for having a look <a class="mention" href="/u/iffix">@iffiX</a>. Really appreciate it heaps!</p>

Leockl: <p>Many thanks <a class="mention" href="/u/mrshenli">@mrshenli</a>.</p>
<p>Your code looks interesting and I am new to PyTorch, so I will have to investigate it line by line what is it doing to see if it helps in what I want to do.</p>

iffiX: <p>Hello, I am optimizing your code, what’s the usual size of <code>n_samples</code> and <code>n_features</code>?</p>

iffiX: <p>If n_samples * n_features &lt; 1e8, and you have a medium to good GPU (&gt;=GTX1080), then there is no need to split by class, below is the “vectorized” implementation, not tested.</p>
<pre><code class="lang-auto"># new implementation of kronecker
def kronecker(A, B):
    return torch.einsum('nab,ncd-&gt;nacbd', A, B)\
        .view(A.size(0), A.size(1)*B.size(1), A.size(2)*B.size(2))
</code></pre>
<p>Main code, no need to use pools or whatever:</p>
<pre><code class="lang-auto"># according to your code, the shape of `X_prime` should be (n_samples, n_features + 1)
# whether encoding = "amplit" or "stereo"
rows = n_samples
cols = n_features + 1


# you may keep this if "n_samples * n_features" &gt; 1e8
# X_prime_class = X_prime[y_class_index == i]

# Number of rows/columns in density matrix
density_nrow_ncol = (n + 1)**self.n_copies

# Encode vectors into quantum densities
# density: [rows, cols, cols], each density[k] is the original `density_each_row`
density = torch.matmul(X_prime.view(rows, cols, 1),
                       X_prime.view(rows, 1, cols))

# Calculate n-fold Kronecker tensor product
if self.n_copies == 1:
    density = density
else:
    density_copy = density
    for u in range(self.n_copies - 1):
        density = kronecker(density, density_copy)

# Calculate sum of quantum densities belonging to either class, per subset split
density_sum = density.sum(dim = 0)

# calculate centroid and q_hels_obs_terms
centroid = (1 / (m_class + 1e-6))*density_sum_class

if self.class_wgt == 'equi':
    q_hels_obs_terms = 0.5*centroid_class
elif self.class_wgt == 'weighted':
    q_hels_obs_terms = (m_class / m)*centroid_class
else:
    raise ValueError('class_wgt should be "equi" or "weighted"')
</code></pre>

Leockl: <p>Hi <a class="mention" href="/u/iffix">@iffiX</a>,</p>
<p>The usual size of <code>n_samples</code> and <code>n_features</code> can be anything really, since I have written a classifier that can be used for any general datasets. But I will take note of your rule of thumb about if “n_samples * n_features &lt; 1e8, then no need to split by class”.</p>
<p>Yes you are correct, <code>X_prime</code> is always (n_samples, n_features + 1) regardless of encoding.</p>
<p>I see what you mean now by “vectorize”. First, `torch.matmul()’ itself has the “vectorization” feature where it can multiple vectors inside a matrix in a “vectorization” manner. Second, rather than just using 2d tensors, I can use a 3d (or higher) dimension tensors to incorporate the 2 classes into one tensor object and then vectors inside this one higher dimension tensor object can be “vectorized”.</p>
<p>Questions:</p>
<ol>
<li>Can I ask how did you determine the value <code>1e8</code> in <code>n_samples * n_features &lt; 1e8</code>?</li>
<li>Can I ask why is there a need to add <code>1e-6</code> in <code>(m_class + 1e-6)</code>?</li>
<li>So for my code, it is difficult to parallelize over multiple GPUs at all? It would be nice if at most I could perhaps just parallelize over the 2 classes on 2 GPUs.</li>
</ol>

iffiX: <ol>
<li>a general estimation from experience, its very rough, for your algorithm it could be anywhere between 1e6 to 1e9 depending on the platform and <code>n_features</code>.</li>
<li>prevent zero devision, if denominator &gt; 0</li>
<li>For multiple gpus, split by class, coarser ganularity splits fits gpus better.</li>
</ol>

Leockl: <p>Sorry what did you mean by “coarser ganularity splits fits gpus better”?</p>

iffiX: <p>Simply: give larger chunks of data to GPU, they are beasts.</p>

Leockl: <p>Ok many thanks once again <a class="mention" href="/u/iffix">@iffiX</a></p>
<p>With parallelizing over the 2 GPUs, I am thinking I guess I can use this:</p>
<pre><code class="lang-auto">device = torch.device("cuda:0")
X1 = X1.to(device)
</code></pre>
<pre><code class="lang-auto">device = torch.device("cuda:1")
X2 = X2.to(device)
</code></pre>

Leockl: <p>Hi <a class="mention" href="/u/iffix">@iffiX</a>, really sorry to bother you but would be great if I could get your input on one more question.</p>
<p>If I were to follow the code that you had suggested, where every row vector (of the dataset, or more specifically of  <code>X_prime</code> ) is converted to a matrix (ie. a density) and then these matrices are all put into the one tensor object, I think this will run into memory blow-out issues. For eg., if my dataset has say 200,000 rows, then I will have 200,000 matrices that needs to be stored at one time, and this could cause a memory blow-out.</p>
<p>In comparison, the code that I have calculates the running sum of the matrices/densities at every step when a new row vector is converted to a matrix, therefore I do not need to store all the matrices in one tensor object (and then only sum them all up after).</p>
<p>Just wanted to get your thoughts on this. It’s ok too if you’re unsure.</p>
<p>Many thanks once again.</p>