[BUG] Elastic cannot kill all subprocesses after sending sigterm.
Labels: oncall: distributed, module: multiprocessing, triaged, module: elastic

### 🐛 Describe the bug

```python
def _close(self, death_sig: signal.Signals, timeout: int = 30) -> None:
        if not self.subprocess_handlers:
            return
        for handler in self.subprocess_handlers.values():
            if handler.proc.poll() is None:
                log.warning(
                    "Sending process %s closing signal %s", handler.proc.pid, death_sig.name
                )
                handler.close(death_sig=death_sig)
        end = time.monotonic() + timeout
        for handler in self.subprocess_handlers.values():
            time_to_wait = end - time.monotonic()
            if time_to_wait <= 0:
                break
            try:
                handler.proc.wait(time_to_wait)
            except subprocess.TimeoutExpired:
                # Ignore the timeout expired exception, since
                # the child process will be forcefully terminated via SIGKILL
                pass
        for handler in self.subprocess_handlers.values():
            if handler.proc.poll() is None:
                log.warning(
                    "Unable to shutdown process %s via %s, forcefully exiting via %s",
                    handler.proc.pid, death_sig, _get_kill_signal()
                )
                handler.close(death_sig=_get_kill_signal())
                handler.proc.wait()
```
The main process has timed out for 30 seconds and has not been terminated by funcition 'handler.close'. Using the sigkill to terminate the main process will cause the ppid of the child process under the main process to change to 1, making it impossible to completely clean up, causing the remaining processes to occupy resources. In some third-party device scenarios, the training script cannot be launched for the second time. Only after using pkill can work. Thus, I offer a solution which will record pids of all subprocesses and check whether all the subprocesses have been killed, if not, sigkill will be send to those subprocesses.
```python
import psutil
def get_child_processes(parent_pid):
    parent = psutil.Process(parent_pid)
    children = parent.children(recursive=True)
    return children

#before kill
parent_pid = os.getpid()
parent_pid = handler.proc.pid
child_processes = []
child_processes.append(get_child_processes(parent_pid))

#after kill
for children in child_processes:
    for child in children:
        if child.poll() is None:
            os.kill(child.pid, _get_kill_signal())
```

### Versions

[pip3] flake8==6.0.0
[pip3] flake8-bugbear==23.3.23
[pip3] flake8-comprehensions==3.11.1
[pip3] flake8-executable==2.1.3
[pip3] flake8-logging-format==0.9.0
[pip3] flake8-pyi==23.3.1
[pip3] flake8-simplify==0.19.3
[pip3] mypy==0.960
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.23.1
[pip3] torch==2.1.0.dev20230810+cpu
[pip3] torchvision==0.15.1
[conda] numpy 1.23.1 pypi_0 pypi
[conda] torchvision 0.15.1 pypi_0 pypi

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @penguinwu @VitalyFedyunin @dzhulgakov