Repository: mosaicml/composer

Issue:
Resnet benchmark crashed with exit code -6
**Environment**
<!-- Provide the below as necessary -->

- OS: [Ubuntu 20.04]
- Hardware (GPU, or instance type): [AMD Instinct/ROCm 5.1.1]


**To reproduce**

Steps to reproduce the behavior:

1.  Execute the recipe in https://github.com/mosaicml/benchmarks/tree/main/blogs/resnet. Specifically running the recipe: recipes/resnet50_hot.yaml
2. Benchmark runs for several epochs (and seems to be doing well) after which composer prints an error and terminates the run
ERROR:composer.cli.launcher:Rank 2 crashed with exit code -6

Dont see any error or stack trace on why the specific rank exited.  I enabled FileLogger and dont see any error / stack trace there as well. Here is the last few lines of the rank 2 log file that was generated. 

[EPOCH][batch=42500]: { "metrics/eval/Accuracy": 0.7575, "metrics/eval/CrossEntropy": 1.6468, }
[EPOCH][batch=42500]: { "epoch": 68, }
[EPOCH][batch=42500]: { "metrics/eval/Accuracy": 0.7575, "metrics/eval/CrossEntropy": 1.6468, }
[EPOCH][batch=42500]: { "epoch": 68, }
[stderr]: INFO:composer.algorithms.progressive_resizing.progressive_resizing:Applied Progressive Resizing with scale_factor=0.9820359281437125 and mode=resize.
[stderr]: Old input dimensions: (H,W)=(167, 167).
[stderr]: New input dimensions: (H,W)=(164, 164)

Is there a way to enable more logging or understand what the exit code -6 means? 

## Expected behavior

Benchmark runs to completion. 







Discussion:
Hi @gopitk we haven't tested everything on AMD GPUs just yet. If you are using the `composer` command, run with `--stderr` to see the logs. @florescl has been testing on AMD GPUs internally, according to her, you'll likely seen an error like:
```
:0:rocdevice.cpp            
:2603: 7586688823011 us: 75436: [tid:0x7f54541db700] 
Device::callbackQueue aborting with error : HSA_STATUS_ERROR_MEMORY_FAULT: Agent attempted to access an inaccessible address. code: 0x2b
```


@gopitk in order to run with --stderr you can do "composer -n num_gpus --stderr "{local_rank}.log" examples/run_composer_trainer.py -f yaml"
Thanks @florescl  and @hanlint. I am able to get the errors now in the file. (It was a bit confusing to me to not see stderr by default in composer. Maybe something that can be changed?). Anyway this seems like an issue in the underlying ROCm layers. Tagging @sunway513 . 
> Thanks @florescl and @hanlint. I am able to get the errors now in the file. (It was a bit confusing to me to not see stderr by default in composer. Maybe something that can be changed?). Anyway this seems like an issue in the underlying ROCm layers. Tagging @sunway513 .

Hmm, the stdout and stderr error for any crashed process should print by default when using the launcher script. Curious did the error print in the console when using `--stderr "{local_rank}.log`, or did it only print in the file?

Resolving patch:
diff --git a/composer/cli/launcher.py b/composer/cli/launcher.py
--- a/composer/cli/launcher.py
+++ b/composer/cli/launcher.py
@@ -413,10 +413,14 @@ def _cleanup_processes(processes: Dict[int, subprocess.Popen]):
                         pass
     for global_rank, process in processes.items():
         process.poll()
-        if process.returncode is not None and process.returncode > 0:
+        if process.returncode is not None and process.returncode != 0:
+            if -process.returncode in (signal.SIGKILL, signal.SIGTERM):
+                # Negative return codes indicate the process was killed via a signal
+                # If the launcher script killed the training process (which would happen via SIGKILL or SIGTERM),
+                # then do not print the stack trace.
+                continue
             # only print the processes that have actually crashed,
-            # not the ones that were killed, which would result in
-            # a negative exit code
+            # not the ones that were killed
             _print_process_exit_status(global_rank, process)
 
 


Test patch: