Can I get help with this error: xmp.spawn() (import torch_xla.distributed.xla_multiprocessing as xmp)

whguswls6471: <p><img src="https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F13099637%2F9dbe6e93fadbfe845a18a5aa90b7a381%2F0002.png?generation=1673716505483417&amp;alt=media" alt="" role="presentation" width="690" height="365"></p>
<p>I tried several ways to fix the bug</p>
<ol>
<li>
<p>trying<br>
<code> !pip install cloud-tpu-client==0.10 torch==1.13.0 https://storage.googleapis.com/tpu-pytorch/wheels/colab/torch_xla-1.13-cp38-cp38-linux_x86_64.whl</code></p>
</li>
<li>
<p>trying<br>
<code> !curl https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py -o pytorch-xla-env-setup.py</code><br>
<code>!python pytorch-xla-env-setup.py </code></p>
</li>
<li>
<p>installing nothing, just import torch_xla</p>
</li>
</ol>
<p>but it all ended to error…<br>
I spended hours to fix this error by searching, but nothing worked.<br>
And its weird that the code works on Colab, but not in Kaggle notebook<br>
I can’t know what I missed</p>
<p>I will add the log of the error at the comments.<br>
Thank you.</p>

whguswls6471: <p>WARNING: Logging before InitGoogle() is written to STDERR<br>
I0000 00:00:1673716544.649557     269 common_lib.cc:145] Failed to fetch URL on try 1 out of 6: Timeout was reached<br>
I0000 00:00:1673716555.150005     269 common_lib.cc:145] Failed to fetch URL on try 2 out of 6: Timeout was reached<br>
I0000 00:00:1673716565.651888     269 common_lib.cc:145] Failed to fetch URL on try 3 out of 6: Timeout was reached<br>
I0000 00:00:1673716576.152164     269 common_lib.cc:145] Failed to fetch URL on try 4 out of 6: Timeout was reached<br>
I0000 00:00:1673716586.653446     269 common_lib.cc:145] Failed to fetch URL on try 5 out of 6: Timeout was reached<br>
I0000 00:00:1673716597.153993     269 common_lib.cc:145] Failed to fetch URL on try 6 out of 6: Timeout was reached<br>
Failed to get agent-worker-number with error: Timeout was reachedI0000 00:00:1673716607.655526     269 common_lib.cc:145] Failed to fetch URL on try 1 out of 6: Timeout was reached<br>
I0000 00:00:1673716618.155990     269 common_lib.cc:145] Failed to fetch URL on try 2 out of 6: Timeout was reached<br>
I0000 00:00:1673716628.657651     269 common_lib.cc:145] Failed to fetch URL on try 3 out of 6: Timeout was reached<br>
I0000 00:00:1673716639.157863     269 common_lib.cc:145] Failed to fetch URL on try 4 out of 6: Timeout was reached<br>
I0000 00:00:1673716649.659285     269 common_lib.cc:145] Failed to fetch URL on try 5 out of 6: Timeout was reached<br>
I0000 00:00:1673716660.159948     269 common_lib.cc:145] Failed to fetch URL on try 6 out of 6: Timeout was reached<br>
I0000 00:00:1673716670.661610     269 common_lib.cc:145] Failed to fetch URL on try 1 out of 6: Timeout was reached<br>
I0000 00:00:1673716681.162626     269 common_lib.cc:145] Failed to fetch URL on try 2 out of 6: Timeout was reached<br>
I0000 00:00:1673716691.664552     269 common_lib.cc:145] Failed to fetch URL on try 3 out of 6: Timeout was reached<br>
I0000 00:00:1673716702.164761     269 common_lib.cc:145] Failed to fetch URL on try 4 out of 6: Timeout was reached<br>
I0000 00:00:1673716712.666079     269 common_lib.cc:145] Failed to fetch URL on try 5 out of 6: Timeout was reached<br>
I0000 00:00:1673716723.166580     269 common_lib.cc:145] Failed to fetch URL on try 6 out of 6: Timeout was reached<br>
I0000 00:00:1673716733.667981     269 common_lib.cc:145] Failed to fetch URL on try 1 out of 6: Timeout was reached<br>
I0000 00:00:1673716744.168007     269 common_lib.cc:145] Failed to fetch URL on try 2 out of 6: Timeout was reached<br>
I0000 00:00:1673716754.668983     269 common_lib.cc:145] Failed to fetch URL on try 3 out of 6: Timeout was reached<br>
I0000 00:00:1673716765.169728     269 common_lib.cc:145] Failed to fetch URL on try 4 out of 6: Timeout was reached<br>
I0000 00:00:1673716775.671754     269 common_lib.cc:145] Failed to fetch URL on try 5 out of 6: Timeout was reached<br>
I0000 00:00:1673716786.172612     269 common_lib.cc:145] Failed to fetch URL on try 6 out of 6: Timeout was reached<br>
I0000 00:00:1673716796.673994     269 common_lib.cc:145] Failed to fetch URL on try 1 out of 6: Timeout was reached<br>
I0000 00:00:1673716807.174525     269 common_lib.cc:145] Failed to fetch URL on try 2 out of 6: Timeout was reached<br>
I0000 00:00:1673716817.676271     269 common_lib.cc:145] Failed to fetch URL on try 3 out of 6: Timeout was reached<br>
I0000 00:00:1673716828.177038     269 common_lib.cc:145] Failed to fetch URL on try 4 out of 6: Timeout was reached<br>
Exception in device=TPU:1: tensorflow/compiler/xla/xla_client/mesh_service.cc:329 : Check failed: impl_-&gt;channel-&gt;WaitForConnected( std::chrono::system_clock::now() + std::chrono::seconds(connect_wait_seconds))<br>
*** Begin stack trace ***<br>
tsl::CurrentStackTrace()<br>
xla::service::MeshClient::MeshClient(std::string const&amp;)<br>
xla::service::MeshClient::Get()<br>
xla::ComputationClient::Create()<br>
xla::ComputationClient::Get()</p>
<pre><code>PyCFunction_Call
_PyObject_MakeTpCall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall

_PyObject_GenericGetAttrWithDict
_PyEval_EvalFrameDefault
_PyEval_EvalCodeWithName
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault

_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
PyVectorcall_Call
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
PyVectorcall_Call
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyEval_EvalCodeWithName
_PyFunction_Vectorcall

_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyObject_FastCallDict
_PyObject_Call_Prepend


_PyObject_MakeTpCall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyEval_EvalCodeWithName
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyEval_EvalCodeWithName
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyEval_EvalCodeWithName
PyEval_EvalCodeEx
PyEval_EvalCode


_PyEval_EvalFrameDefault

_PyEval_EvalFrameDefault

_PyEval_EvalFrameDefault


_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyEval_EvalCodeWithName
_PyFunction_Vectorcall

PyVectorcall_Call
_PyEval_EvalFrameDefault
_PyEval_EvalCodeWithName
_PyFunction_Vectorcall

_PyEval_EvalFrameDefault

_PyEval_EvalFrameDefault

_PyEval_EvalFrameDefault

_PyEval_EvalFrameDefault

_PyEval_EvalFrameDefault



_PyObject_MakeTpCall


PyVectorcall_Call
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyEval_EvalCodeWithName
_PyFunction_Vectorcall

_PyEval_EvalFrameDefault
_PyEval_EvalCodeWithName
PyEval_EvalCodeEx
PyEval_EvalCode


_PyEval_EvalFrameDefault
_PyEval_EvalCodeWithName
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyEval_EvalCodeWithName
_PyFunction_Vectorcall
PyVectorcall_Call

Py_RunMain
Py_BytesMain
</code></pre>
<p>*** End stack trace ***<br>
Failed to connect to client mesh master: 785db6c3ea80:57619<br>
Traceback (most recent call last):<br>
File “/usr/local/lib/python3.8/site-packages/torch_xla/distributed/xla_multiprocessing.py”, line 331, in _mp_start_fn<br>
<em>start_fn(index, pf_cfg, fn, args)<br>
Exception in device=TPU:2: tensorflow/compiler/xla/xla_client/mesh_service.cc:329 : Check failed: impl</em>-&gt;channel-&gt;WaitForConnected( std::chrono::system_clock::now() + std::chrono::seconds(connect_wait_seconds))<br>
*** Begin stack trace ***<br>
tsl::CurrentStackTrace()<br>
xla::service::MeshClient::MeshClient(std::string const&amp;)<br>
xla::service::MeshClient::Get()<br>
xla::ComputationClient::Create()</p>
<pre><code>xla::ComputationClient::Get()


PyCFunction_Call
_PyObject_MakeTpCall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall

_PyObject_GenericGetAttrWithDict
_PyEval_EvalFrameDefault
_PyEval_EvalCodeWithName
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault

_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
PyVectorcall_Call
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
PyVectorcall_Call
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyEval_EvalCodeWithName
_PyFunction_Vectorcall

_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyObject_FastCallDict
_PyObject_Call_Prepend


_PyObject_MakeTpCall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyEval_EvalCodeWithName
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyEval_EvalCodeWithName
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyEval_EvalCodeWithName
PyEval_EvalCodeEx
PyEval_EvalCode


_PyEval_EvalFrameDefault

_PyEval_EvalFrameDefault

_PyEval_EvalFrameDefault


_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyEval_EvalCodeWithName
_PyFunction_Vectorcall

PyVectorcall_Call
_PyEval_EvalFrameDefault
_PyEval_EvalCodeWithName
_PyFunction_Vectorcall

_PyEval_EvalFrameDefault

_PyEval_EvalFrameDefault

_PyEval_EvalFrameDefault

_PyEval_EvalFrameDefault

_PyEval_EvalFrameDefault



_PyObject_MakeTpCall


PyVectorcall_Call
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyEval_EvalCodeWithName
_PyFunction_Vectorcall

_PyEval_EvalFrameDefault
_PyEval_EvalCodeWithName
PyEval_EvalCodeEx
PyEval_EvalCode


_PyEval_EvalFrameDefault
_PyEval_EvalCodeWithName
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyEval_EvalCodeWithName
_PyFunction_Vectorcall
PyVectorcall_Call

Py_RunMain
Py_BytesMain
</code></pre>
<p>*** End stack trace ***<br>
Failed to connect to client mesh master: 785db6c3ea80:57619  File “/usr/local/lib/python3.8/site-packages/torch_xla/distributed/xla_multiprocessing.py”, line 324, in _start_fn<br>
_setup_replication()</p>
<p>File “/usr/local/lib/python3.8/site-packages/torch_xla/distributed/xla_multiprocessing.py”, line 316, in _setup_replication<br>
device = xm.xla_device()<br>
File “/usr/local/lib/python3.8/site-packages/torch_xla/core/xla_model.py”, line 245, in xla_device<br>
devices = get_xla_supported_devices(devkind=devkind)<br>
Traceback (most recent call last):<br>
File “/usr/local/lib/python3.8/site-packages/torch_xla/core/xla_model.py”, line 139, in get_xla_supported_devices<br>
xla_devices = _DEVICES.value<br>
File “/usr/local/lib/python3.8/site-packages/torch_xla/distributed/xla_multiprocessing.py”, line 331, in _mp_start_fn<br>
_start_fn(index, pf_cfg, fn, args)<br>
File “/usr/local/lib/python3.8/site-packages/torch_xla/utils/utils.py”, line 32, in value<br>
self._value = self._gen_fn()<br>
File “/usr/local/lib/python3.8/site-packages/torch_xla/distributed/xla_multiprocessing.py”, line 324, in _start_fn<br>
_setup_replication()<br>
File “/usr/local/lib/python3.8/site-packages/torch_xla/core/xla_model.py”, line 21, in <br>
_DEVICES = xu.LazyProperty(lambda: torch_xla._XLAC._xla_get_devices())<br>
File “/usr/local/lib/python3.8/site-packages/torch_xla/distributed/xla_multiprocessing.py”, line 316, in <em>setup_replication<br>
device = xm.xla_device()<br>
RuntimeError: tensorflow/compiler/xla/xla_client/mesh_service.cc:329 : Check failed: impl</em>-&gt;channel-&gt;WaitForConnected( std::chrono::system_clock::now() + std::chrono::seconds(connect_wait_seconds))<br>
*** Begin stack trace ***<br>
tsl::CurrentStackTrace()<br>
xla::service::MeshClient::MeshClient(std::string const&amp;)<br>
xla::service::MeshClient::Get()<br>
xla::ComputationClient::Create()</p>
<pre><code>xla::ComputationClient::Get()


PyCFunction_Call
_PyObject_MakeTpCall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall

_PyObject_GenericGetAttrWithDict
_PyEval_EvalFrameDefault
_PyEval_EvalCodeWithName
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault

_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
PyVectorcall_Call
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
PyVectorcall_Call
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyEval_EvalCodeWithName
_PyFunction_Vectorcall

_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyObject_FastCallDict
_PyObject_Call_Prepend


_PyObject_MakeTpCall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyEval_EvalCodeWithName
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyEval_EvalCodeWithName
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyEval_EvalCodeWithName
PyEval_EvalCodeEx
PyEval_EvalCode


_PyEval_EvalFrameDefault

_PyEval_EvalFrameDefault

_PyEval_EvalFrameDefault


_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyEval_EvalCodeWithName
_PyFunction_Vectorcall

PyVectorcall_Call
_PyEval_EvalFrameDefault
_PyEval_EvalCodeWithName
_PyFunction_Vectorcall

_PyEval_EvalFrameDefault

_PyEval_EvalFrameDefault

_PyEval_EvalFrameDefault

_PyEval_EvalFrameDefault

_PyEval_EvalFrameDefault



_PyObject_MakeTpCall


PyVectorcall_Call
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyEval_EvalCodeWithName
_PyFunction_Vectorcall

_PyEval_EvalFrameDefault
_PyEval_EvalCodeWithName
PyEval_EvalCodeEx
PyEval_EvalCode


_PyEval_EvalFrameDefault
_PyEval_EvalCodeWithName
_PyFunction_Vectorcall
_PyEval_EvalFrameDefault
_PyEval_EvalCodeWithName
_PyFunction_Vectorcall
PyVectorcall_Call

Py_RunMain
Py_BytesMain
</code></pre>
<h2>
<a name="end-stack-trace-failed-to-connect-to-client-mesh-master-785db6c3ea8057619-1" class="anchor" href="#end-stack-trace-failed-to-connect-to-client-mesh-master-785db6c3ea8057619-1"></a>*** End stack trace ***<br>
Failed to connect to client mesh master: 785db6c3ea80:57619</h2>
<p><img src="https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F13099637%2F798a27ea0fe98be664d065a9f421280f%2F0003.png?generation=1673718190542032&amp;alt=media" alt="" role="presentation" width="690" height="414"><br>
ProcessExitedException                    Traceback (most recent call last)<br>
Cell In[4], line 22<br>
20 flags = {}<br>
21 # Note: Colab only supports start_method=‘fork’<br>
—&gt; 22 xmp.spawn(simple_map_fn, args=(flags,), nprocs=8, start_method=‘fork’)</p>
<p>File /usr/local/lib/python3.8/site-packages/torch_xla/distributed/xla_multiprocessing.py:393, in spawn(fn, args, nprocs, join, daemon, start_method)<br>
391   _start_fn(0, pf_cfg, fn, args)<br>
392 else:<br>
 → 393   return torch.multiprocessing.start_processes(<br>
394       _mp_start_fn,<br>
395       args=(pf_cfg, fn, args),<br>
396       nprocs=pf_cfg.num_devices,<br>
397       join=join,<br>
398       daemon=daemon,<br>
399       start_method=start_method)</p>
<p>File /usr/local/lib/python3.8/site-packages/torch/multiprocessing/spawn.py:198, in start_processes(fn, args, nprocs, join, daemon, start_method)<br>
195     return context<br>
197 # Loop on join until it returns True or raises an exception.<br>
 → 198 while not context.join():<br>
199     pass</p>
<p>File /usr/local/lib/python3.8/site-packages/torch/multiprocessing/spawn.py:149, in ProcessContext.join(self, timeout)<br>
140         raise ProcessExitedException(<br>
141             “process %d terminated with signal %s” %<br>
142             (error_index, name),<br>
(…)<br>
146             signal_name=name<br>
147         )<br>
148     else:<br>
 → 149         raise ProcessExitedException(<br>
150             “process %d terminated with exit code %d” %<br>
151             (error_index, exitcode),<br>
152             error_index=error_index,<br>
153             error_pid=failed_process.pid,<br>
154             exit_code=exitcode<br>
155         )<br>
157 original_trace = self.error_queues[error_index].get()<br>
158 msg = “\n\n– Process %d terminated with the following error:\n” % error_index</p>
<p>ProcessExitedException: process 1 terminated with exit code 17</p>

whguswls6471: <p>this is a other error when I run the cell below</p>
<p><img src="https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F13099637%2F55108edf4a2f382b2e48bf7f87012be1%2F0005.png?generation=1673718289545432&amp;alt=media" alt="" role="presentation" width="514" height="175"></p>

whguswls6471: <p><img src="https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F13099637%2F60ffef10ecc67daafbe3cd1186e40934%2F0004.png?generation=1673718251077681&amp;alt=media" alt="" role="presentation" width="369" height="500"></p>

Jack_Cao: <p>Thanks! Do you mind opening a github issue in <a href="https://github.com/pytorch/xla/issues" class="inline-onebox" rel="noopener nofollow ugc">Issues · pytorch/xla · GitHub</a>(we checked issues regularly but less frequent for pytorch discuss)? We will help you there.</p>