Repository: European-XFEL/EXtra-data

Issue:
RunDirectory does not work as context manager when voview file present
[In the Jungfrau correction notebook, the raw run files are opened using a context manager](https://git.xfel.eu/detectors/pycalibration/-/blob/master/notebooks/Jungfrau/Jungfrau_Gain_Correct_and_Verify_NBC.ipynb#L697).  

This works fine when no voview files were created, as the files are opened directly.  
It's also fine to use the voview file when using `RunDirectory` as a regular object.  
However, when opening a run using a voview file ([the default behaviour when available](https://github.com/European-XFEL/EXtra-data/blob/d4c547c3af9a993ec2da41b5bf3d830dce9ab43e/extra_data/reader.py#L1398:L1401)), an [error is raised](https://github.com/European-XFEL/EXtra-data/blob/master/extra_data/reader.py#L186:L194) as files cannot be closed when leaving the context.  

We noticed this recently as HED is using the RunValidator which creates such files, whereas other instruments do not (yet).

## How to Reproduce  

This can be reproduced with the following:  

```python
In [15]: from extra_data import RunDirectory
    ...: with RunDirectory("/gpfs/exfel/exp/HED/202330/p900338/raw/r0052", "*S00000*") as
    ...:  fd:
    ...:     print(fd)
    ...: 
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
<ipython-input-15-53c99aa5c4ec> in <module>
      1 from extra_data import RunDirectory
----> 2 with RunDirectory("/gpfs/exfel/exp/HED/202330/p900338/raw/r0052", "*S00000*") as fd:
      3     print(fd)
      4 

~/EXtra-data/extra_data/reader.py in __enter__(self)
    186     def __enter__(self):
    187         if not self.ctx_closes:
--> 188             raise Exception(
    189                 "Only DataCollection objects created by opening "
    190                 "files directly can be used in a 'with' statement, "

Exception: Only DataCollection objects created by opening files directly can be used in a 'with' statement, not those created by selecting from or merging others.

```  

## Possible fixes  

The current behaviour is not necessarily wrong, as it's difficult to know before hand if the `DataCollection` is going to be used as a context manager.

In the case of the correction, [we explicitly ignore the voview file](https://git.xfel.eu/detectors/pycalibration/-/merge_requests/793) as we want to simplify file access.  

I think it would be beneficial to clarify the error message and suggest to pass `_use_voview=False`.  

Another solution would be to [remove the check in `__enter__`](https://github.com/European-XFEL/EXtra-data/blob/d4c547c3af9a993ec2da41b5bf3d830dce9ab43e/extra_data/reader.py#L187) as the check whether to close files is done again  in [`__exit__`](https://github.com/European-XFEL/EXtra-data/blob/d4c547c3af9a993ec2da41b5bf3d830dce9ab43e/extra_data/reader.py#L198).  

I'm happy to contribute a solution, but would need guidance.


Thanks,
Cyril


Comments:

Patch:
diff --git a/extra_data/reader.py b/extra_data/reader.py
index 21ad0f4..90643e5 100644
--- a/extra_data/reader.py
+++ b/extra_data/reader.py
@@ -1398,7 +1398,9 @@ def RunDirectory(
     if _use_voview and (sel_files == files):
         voview_file_acc = voview.find_file_valid(path)
         if voview_file_acc is not None:
-            return DataCollection([voview_file_acc], is_single_run=True)
+            return DataCollection([voview_file_acc],
+                                  is_single_run=True,
+                                  ctx_closes=True)
 
     files_map = RunFilesMap(path)
     t0 = time.monotonic()
diff --git a/setup.py b/setup.py
index 4237df0..1e974b9 100755
--- a/setup.py
+++ b/setup.py
@@ -76,6 +76,7 @@ setup(name="EXtra-data",
               'pytest',
               'pytest-cov',
               'testpath',
+              'psutil',
           ]
       },
       python_requires='>=3.6',


Test patch:
diff --git a/extra_data/tests/test_voview.py b/extra_data/tests/test_voview.py
index 3f0bb96..3179802 100644
--- a/extra_data/tests/test_voview.py
+++ b/extra_data/tests/test_voview.py
@@ -35,6 +35,10 @@ def test_use_voview(mock_spb_raw_run, tmp_path):
     assert 'SPB_DET_AGIPD1M-1/DET/0CH0:xtdf' in run.instrument_sources
     assert 'SA1_XTD2_XGM/DOOCS/MAIN' in run.control_sources
 
+    with RunDirectory(str(new_run_dir)) as run:
+        assert 'SPB_DET_AGIPD1M-1/DET/0CH0:xtdf' in run.instrument_sources
+        assert 'SA1_XTD2_XGM/DOOCS/MAIN' in run.control_sources
+
 
 def open_run_with_voview(run_src, new_run_dir):
     copytree(run_src, new_run_dir)