Potential issue of "errno: 98- Address already in use" error in DDP (with torchrun)

username: duan22677
cooked: <p>During the use of torch run (with ddp), sometimes there may be random occurrences of ‘errno: 98- Address already in use’, for example:</p>
<pre data-code-wrap="bash"><code class="lang-bash">[W socket.cpp:436] [c10d] The server socket has failed to bind to [::]:29400 (errno: 98 - Address already in use).
[W socket.cpp:436] [c10d] The server socket has failed to bind to 0.0.0.0:29400 (errno: 98 - Address already in use).
[E socket.cpp:472] [c10d] The server socket has failed to listen on any local network address.
</code></pre>
<p>There is a similar issue on Github: <a href="https://github.com/pytorch/pytorch/issues/85604" rel="noopener nofollow ugc">issue 85604</a></p>
<p>Both the issue 85604 mentioned above or other answers searched online, the <strong>explanations</strong> provided are mostly:</p>
<ul>
<li>“Perhaps the previous pytorch training task was not completely exited, master_port is still being occupied”</li>
<li>“(for unknown reasons), port is being occupied”.</li>
</ul>
<p>And The <strong>solution</strong> is : “provide a new and free master-port”</p>
<p>When I was reading the source code of pytorch torch, I found that there were two places where TCPStore and master port were used.</p>
<ol>
<li><a href="https://github.com/pytorch/pytorch/blob/orig/release/2.3/torch/distributed/elastic/rendezvous/registry.py#L36" rel="noopener nofollow ugc">create_c10_backend</a>, where using master post in and creating the first <a href="https://github.com/pytorch/pytorch/blob/orig/release/2.3/torch/distributed/elastic/rendezvous/c10d_rendezvous_backend.py#L157-L163" rel="noopener nofollow ugc">TCPStore</a>.</li>
</ol>
<p>In my options, the TCPStore is used to store Rendezvous information. It’s only used to synchronize the status information of multiple ElasticAgents and is not actually related to DDP.</p>
<p>There may be a problem with the ‘Address already in use’ mentioned above. I can find a free port and solve it by specifying <code>--rdzv_endpoint=${host_ip}:{free_port}</code>.</p>
<p>And after this first TCPStore is created, in <a href="https://github.com/pytorch/pytorch/blob/orig/release/2.3/torch/distributed/elastic/agent/server/api.py#L557-L563" rel="noopener nofollow ugc">_initialize_workers</a>, a new <code>master-port</code> is find by <a href="https://github.com/pytorch/pytorch/blob/orig/release/2.3/torch/distributed/elastic/agent/server/api.py#L359-L387" rel="noopener nofollow ugc">_get_socket_with_port</a>(and this <code>master-port</code> is unable to specify by passing args parameters or environment variables outside. And later, two key values, <code>MASTER_ADDR</code> and <code>MASTER_PORT</code> (as environment variable?), will be stored in the first TCPStore, see <a href="https://github.com/pytorch/pytorch/blob/orig/release/2.3/torch/distributed/elastic/agent/server/api.py#L510C1-L529C76" rel="noopener nofollow ugc">_set_master_addr_port</a>.</p>
<ol start="2">
<li>The second creation of TCPStore is in dist <a href="https://github.com/pytorch/pytorch/blob/orig/release/2.3/torch/distributed/distributed_c10d.py#L1302-L1310" rel="noopener nofollow ugc">init_process_group</a>, which finally call <a href="https://github.com/pytorch/pytorch/blob/orig/release/2.3/torch/distributed/rendezvous.py#L207-L251" rel="noopener nofollow ugc">_env_rendezvous_handler</a> to create the TCPStore. And in  <code>_env_rendezvous_handler</code>:</li>
</ol>
<pre data-code-wrap="python"><code class="lang-python">def _env_rendezvous_handler(
    url: str, timeout: timedelta = default_pg_timeout, **kwargs
):
    ...
    
    # find master_addr and master_port from environment variable
    master_addr = _get_env_or_raise("MASTER_ADDR")
    master_port = int(_get_env_or_raise("MASTER_PORT"))
    ...
    
    # 
    store = _create_c10d_store(master_addr, master_port, rank, world_size, timeout, use_libuv)

    yield (store, rank, world_size)

</code></pre>
<p>We can see that the second TCPStore in DDP will default to use the  <code>MASTER_PORT</code> which is found by socket and set to environment before.</p>
<p>And there may also be <code>errno: 98- Address already in use</code> issues in <code>_create_c10d_store</code>.</p>
<p>There may be the following reasons:</p>
<ol>
<li>It is possible that the <code>master-port</code> in the <code>_set_master_addr_port</code> has not been fully released, perhaps due to TIMP_WAIT? I’m not quite sure. But it maybe cause the error when using the same <code>master-port</code> in <code>_create_c10d_store</code>;</li>
<li>The <code>master-port</code> in the <code>_set_master_addr_port</code> has been completely released, but it has been occupied by other processes in the system before being reused in <code>_create_c10d_store</code>, which can also cause problems.</li>
</ol>
<p>To verify the reason, I conducted a test. Use both <code>socket.bind</code> , <code>ss -tlnp | grep &lt;port&gt;</code>, <code>lsof -i:&lt;port&gt;</code> to check if the port is occupied, like in <code>_create_c10d_store</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">
def is_port_in_use(port, host='127.0.0.1', rank=-1):  
            
    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:  
        try:  
            s.bind((host, port))  
            in_use = False  
        except socket.error as e:
            print(f"---&gt; socket error: {e}")
            in_use = True
                    
    if in_use:
        print(f"===&gt; rank {rank}, socket check, port {port} is in use")
    else:
        print(f"===&gt; rank {rank}, socket check, port {port} is in free")

def _create_c10d_store(hostname, port, rank, world_size, timeout, use_libuv=False) -&gt; Store:

    if not 0 &lt;= port &lt; 2**16:
        raise ValueError(f"port must have value from 0 to 65535 but was {port}.")

    if _torchelastic_use_agent_store():
        attempt = os.environ["TORCHELASTIC_RESTART_COUNT"]
        tcp_store = TCPStore(hostname, port, world_size, False, timeout)
        return PrefixStore(f"/worker/attempt_{attempt}", tcp_store)
    else:
        start_daemon = rank == 0
        
        # check
        is_port_in_use(port, hostname, rank)
        
        return TCPStore(
            hostname, port, world_size, start_daemon, timeout, multi_tenant=True, use_libuv=use_libuv
        )
</code></pre>
<p>The test results are as follows:</p>
<ul>
<li>port free, and the running was successful:</li>
</ul>
<pre data-code-wrap="bash"><code class="lang-bash">===&gt; rank 0, socket check, port 39995 is in free
===&gt; rank 1, socket check, port 39995 is in free

# no error, success
</code></pre>
<ul>
<li>port in use, running fail</li>
</ul>
<pre data-code-wrap="bash"><code class="lang-bash">===&gt; rank 0, socket check, port 48763 is in use
===&gt; rank 1, socket check, port 48763 is in use

# failed
[W socket.cpp:436] [c10d] The server socket has failed to bind to [::]:48763 (errno: 98 - Address already in use).
[W socket.cpp:436] [c10d] The server socket has failed to bind to 0.0.0.0:48763 (errno: 98 - Address already in use).
[E socket.cpp:472] [c10d] The server socket has failed to listen on any local network address.
</code></pre>
<ul>
<li>port in use, running successful (confused???)</li>
</ul>
<pre data-code-wrap="bash"><code class="lang-bash">===&gt; rank 0, socket check, port 37956 is in use
===&gt; rank 1, socket check, port 37956 is in use

# no error, success
</code></pre>
<p>I have the following questions:</p>
<p><strong>Question1:</strong><br>
What is the true reason for the <code>errno: 98- Address already in use</code> issues in <code>_create_c10d_store</code>?</p>
<p><strong>Question2</strong>:<br>
Why is the <code>socket.bind</code> detection port being used, but TCPStore using the same port does not raise error? (Is there something wrong with my testing method or did I overlook anything?)</p>
<p><strong>Question3</strong>:<br>
How to solve or avoid this problem from an engineering perspective? (Instead of running again after discovering errors)</p>