Repository: cta-observatory/lstmcpipe

Issue:
Merging failure does not make job fail
>   @vuillaut It seems other base productions have the same issue:
> `20221215_v0.9.12_base_prod` and `20230127_v0.9.12_base_prod_az_tel`

_Originally posted by @SeiyaNozaki in https://github.com/cta-observatory/lstmcpipe/issues/371#issuecomment-1446600468_
            

See  e.g.
`/fefs/aswg/data/mc/DL1/AllSky/20221215_v0.9.12_base_prod/TrainingDataset/dec_931/Protons/merging-output.e`

``` 
Traceback (most recent call last):
  File "/fefs/aswg/software/conda/envs/lstchain-v0.9.12/bin/lstchain_merge_hdf5_files", line 8, in <module>
    sys.exit(main())
  File "/fefs/aswg/software/conda/envs/lstchain-v0.9.12/lib/python3.8/site-packages/lstchain/scripts/lstchain_merge_hdf5_files.py", line 88, in main
    auto_merge_h5files(
  File "/fefs/aswg/software/conda/envs/lstchain-v0.9.12/lib/python3.8/site-packages/lstchain/io/io.py", line 341, in auto_merge_h5files
    out_node.append(in_node.read().astype(out_node.dtype))
ValueError: structures must have the same size
```



Discussion:
codecov[bot]: ## [Codecov](https://codecov.io/gh/cta-observatory/lstmcpipe/pull/398?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=cta-observatory) Report
Patch coverage: **`100.00`**% and project coverage change: **`+0.05`** :tada:
> Comparison is base [(`825c042`)](https://codecov.io/gh/cta-observatory/lstmcpipe/commit/825c042148c211e0581d5ae4068b6d3d296debb3?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=cta-observatory) 30.60% compared to head [(`e2b0c6b`)](https://codecov.io/gh/cta-observatory/lstmcpipe/pull/398?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=cta-observatory) 30.65%.

> :exclamation: Current head e2b0c6b differs from pull request most recent head 14be2e7. Consider uploading reports for the commit 14be2e7 to get more accurate results

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##           master     #398      +/-   ##
==========================================
+ Coverage   30.60%   30.65%   +0.05%     
==========================================
  Files          46       46              
  Lines        2774     2776       +2     
==========================================
+ Hits          849      851       +2     
  Misses       1925     1925              
```


| [Impacted Files](https://codecov.io/gh/cta-observatory/lstmcpipe/pull/398?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=cta-observatory) | Coverage Δ | |
|---|---|---|
| [lstmcpipe/tests/test\_utils.py](https://codecov.io/gh/cta-observatory/lstmcpipe/pull/398?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=cta-observatory#diff-bHN0bWNwaXBlL3Rlc3RzL3Rlc3RfdXRpbHMucHk=) | `95.94% <100.00%> (ø)` | |
| [lstmcpipe/utils.py](https://codecov.io/gh/cta-observatory/lstmcpipe/pull/398?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=cta-observatory#diff-bHN0bWNwaXBlL3V0aWxzLnB5) | `81.71% <100.00%> (+0.21%)` | :arrow_up: |

Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=cta-observatory). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=cta-observatory)

</details>

[:umbrella: View full report in Codecov by Sentry](https://codecov.io/gh/cta-observatory/lstmcpipe/pull/398?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=cta-observatory).   
:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=cta-observatory).


Patch:
diff --git a/lstmcpipe/utils.py b/lstmcpipe/utils.py
index a122672..6ab3063 100644
--- a/lstmcpipe/utils.py
+++ b/lstmcpipe/utils.py
@@ -256,6 +256,11 @@ class SbatchLstMCStage:
     def compose_wrap_command(self, wrap_command=None, source_env="", backend=""):
         if wrap_command is None or wrap_command == "":
             warnings.warn("You must pass a command to be batched! ")
+        else:
+            # Remove trailing semicolon and any extra spaces
+            wrap_command = wrap_command.rstrip(";").strip()
+            # exit if any command fails
+            wrap_command = wrap_command + ' || exit ' + r'\$?'
         if source_env != "" and not source_env.strip().endswith(";"):
             source_env = f"{source_env.strip()}; "
         if backend != "" and not backend.strip().endswith(";"):


Test patch:
diff --git a/lstmcpipe/tests/test_utils.py b/lstmcpipe/tests/test_utils.py
index 4be15ab..7957757 100644
--- a/lstmcpipe/tests/test_utils.py
+++ b/lstmcpipe/tests/test_utils.py
@@ -6,10 +6,10 @@ from ruamel.yaml import YAML
 from lstmcpipe.utils import rerun_cmd, dump_lstchain_std_config, SbatchLstMCStage, run_command
 
 
-def test_save_log_to_file():
+def test_save_log_to_file(tmp_path):
     from ..utils import save_log_to_file
 
-    outfile_yml = Path('dummy_log.yml')
+    outfile_yml = Path(tmp_path, 'dummy_log.yml')
 
     dummy_log = {'dummy_jobid': 'sbatch --parsable --wrap="sleep 10"'}
     save_log_to_file(dummy_log, outfile_yml)
@@ -74,14 +74,14 @@ def test_sbatch_lst_mc_stage():
     sbatch = SbatchLstMCStage(stage="r0_to_dl1", wrap_command="command to be batched")
     assert (
         sbatch.slurm_command == 'sbatch --parsable --partition=long --job-name=r0_dl1 --array=0-0%100 '
-                                '--error=./slurm-%j.e --output=./slurm-%j.o  --wrap="command to be batched"'
+        '--error=./slurm-%j.e --output=./slurm-%j.o  --wrap="command to be batched' + ' || exit ' + r'\$?"'
     )
 
     sbatch.extra_slurm_options = {'account': 'lstrta', 'partition': 'xxl', 'mem': '160G', 'cpus-per-task': 32}
     assert (
         sbatch.slurm_command == 'sbatch --parsable --partition=xxl --job-name=r0_dl1 --array=0-0%100 '
-                                '--error=./slurm-%j.e --output=./slurm-%j.o --account=lstrta --mem=160G '
-                                '--cpus-per-task=32  --wrap="command to be batched"'
+        '--error=./slurm-%j.e --output=./slurm-%j.o --account=lstrta --mem=160G '
+        '--cpus-per-task=32  --wrap="command to be batched' + ' || exit ' + r'\$?"'
     )
 
     sbatch.extra_slurm_options = None
@@ -93,7 +93,10 @@ def test_sbatch_lst_mc_stage():
         source_env="source .bashrc_file; conda activate   ",
         backend="  export MPLBACKEND=Agg    ",
     )
-    assert sbatch.wrap_cmd == '--wrap="export MPLBACKEND=Agg; source .bashrc_file; conda activate; python args"'
+    assert (
+        sbatch.wrap_cmd
+        == '--wrap="export MPLBACKEND=Agg; source .bashrc_file; conda activate; python args' + ' || exit ' + r'\$?"'
+    )
 
     with pytest.raises(ValueError):
         sbatch.submit()  # slurm not installed