Repository: fjosw/pyerrors

Issue:
Files keyword for multiple reps in read_sfcf
Hi there,
just wanted to make you aware (and maybe have a small discussion) about an inconsistency in the `input.sfcf.read_sfcf_multi()` and therefore also in the `input.sfcf.read_sfcfi()` method. There exists a keyword `files`, which takes a list of files to be read. I noticed, that this only works for one replica and not if you have multiple ones.
I can take care of this, and it seems not that hard to me.  I'll make a PR when I am done.  Before that, I just wanted to check in for one thing:
This functionality can be implemented either as a list of lists, or, similar to `idl` as a dict of lists.
There are arguments for both implementations in my opinion. On one hand, this kind of functionality is implemented in this function via lists of lists. On the other hand, I think it is saver to implement it via a `dict`, as one can recheck, if the files are written in the right replica.
It is possible to implement both and a third option which covers the current (false-ish) implementation, such that we do not break any backwards compatibility.

Discussion:
@s-kuberski: Hi,
I don't really understand your comment concerning 'idl'. Do you mean the way this attribute is stored in an `Obs`? I think a list is fine and it is also in line with what is used for the openQCD input routines (where, of course, only one file per replicum is used).@jkuhl-uni: I think we are talking about the same, yes. My alternative to lists would have been to use the same structure as used in `Obs.idl`. However, I agree that this change would change the user API by quite a bit and we can discard the idea.

Solution patch:
diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml
index 9b204da..03ca7c2 100644
--- a/.github/workflows/docs.yml
+++ b/.github/workflows/docs.yml
@@ -11,10 +11,10 @@ jobs:
 
     steps:
       - name: Set up Python environment
-        uses: actions/setup-python@v4
+        uses: actions/setup-python@v5
         with:
           python-version: "3.10"
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
       - name: Updated documentation
         run: |
           git config --global user.email "${{ github.actor }}@users.noreply.github.com"
diff --git a/.github/workflows/examples.yml b/.github/workflows/examples.yml
index dd2fc4c..6763657 100644
--- a/.github/workflows/examples.yml
+++ b/.github/workflows/examples.yml
@@ -21,10 +21,10 @@ jobs:
 
     steps:
       - name: Checkout source
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
 
       - name: Setup python
-        uses: actions/setup-python@v4
+        uses: actions/setup-python@v5
         with:
           python-version: ${{ matrix.python-version }}
 
diff --git a/.github/workflows/flake8.yml b/.github/workflows/flake8.yml
index d931f24..c6625b3 100644
--- a/.github/workflows/flake8.yml
+++ b/.github/workflows/flake8.yml
@@ -13,9 +13,9 @@ jobs:
     name: Lint
     steps:
       - name: Check out source repository
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
       - name: Set up Python environment
-        uses: actions/setup-python@v4
+        uses: actions/setup-python@v5
         with:
           python-version: "3.10"
       - name: flake8 Lint
diff --git a/.github/workflows/pytest.yml b/.github/workflows/pytest.yml
index b6a5290..1bdd412 100644
--- a/.github/workflows/pytest.yml
+++ b/.github/workflows/pytest.yml
@@ -24,10 +24,10 @@ jobs:
 
     steps:
       - name: Checkout source
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
 
       - name: Setup python
-        uses: actions/setup-python@v4
+        uses: actions/setup-python@v5
         with:
           python-version: ${{ matrix.python-version }}
 
@@ -41,6 +41,7 @@ jobs:
           pip install pytest-benchmark
           pip install hypothesis
           pip install py
+          pip install pyarrow
           pip freeze
 
       - name: Run tests
diff --git a/pyerrors/input/sfcf.py b/pyerrors/input/sfcf.py
index 8295755..94aa194 100644
--- a/pyerrors/input/sfcf.py
+++ b/pyerrors/input/sfcf.py
@@ -121,7 +121,7 @@ def read_sfcf_multi(path, prefix, name_list, quarks_list=['.*'], corr_type_list=
         String that separates the ensemble identifier from the configuration number (default 'n').
     replica: list
         list of replica to be read, default is all
-    files: list
+    files: list[list[int]]
         list of files to be read per replica, default is all.
         for non-compact output format, hand the folders to be read here.
     check_configs: list[list[int]]
@@ -236,6 +236,16 @@ def read_sfcf_multi(path, prefix, name_list, quarks_list=['.*'], corr_type_list=
             rep_path = path + '/' + item
             if "files" in kwargs:
                 files = kwargs.get("files")
+                if isinstance(files, list):
+                    if all(isinstance(f, list) for f in files):
+                        files = files[i]
+                    elif all(isinstance(f, str) for f in files):
+                        files = files
+                    else:
+                        raise TypeError("files has to be of type list[list[str]] or list[str]!")
+                else:
+                    raise TypeError("files has to be of type list[list[str]] or list[str]!")
+
             else:
                 files = []
             sub_ls = _find_files(rep_path, prefix, compact, files)
@@ -248,7 +258,7 @@ def read_sfcf_multi(path, prefix, name_list, quarks_list=['.*'], corr_type_list=
                     else:
                         rep_idl.append(int(cfg[3:]))
                 except Exception:
-                    raise Exception("Couldn't parse idl from directroy, problem with file " + cfg)
+                    raise Exception("Couldn't parse idl from directory, problem with file " + cfg)
             rep_idl.sort()
             # maybe there is a better way to print the idls
             if not silent:
@@ -309,7 +319,10 @@ def read_sfcf_multi(path, prefix, name_list, quarks_list=['.*'], corr_type_list=
             w = specs[3]
             w2 = specs[4]
             if "files" in kwargs:
-                ls = kwargs.get("files")
+                if isinstance(kwargs.get("files"), list) and all(isinstance(f, str) for f in kwargs.get("files")):
+                    name_ls = kwargs.get("files")
+                else:
+                    raise TypeError("In append mode, files has to be of type list[str]!")
             else:
                 name_ls = ls
                 for exc in name_ls:


Test patch:
diff --git a/tests/sfcf_in_test.py b/tests/sfcf_in_test.py
index ac05dc0..f92126f 100644
--- a/tests/sfcf_in_test.py
+++ b/tests/sfcf_in_test.py
@@ -50,6 +50,18 @@ def test_o_bi(tmp_path):
     assert f_A[2].value == -41.025094911185185
 
 
+def test_o_bi_files(tmp_path):
+    build_test_environment(str(tmp_path), "o", 10, 3)
+    f_A = sfin.read_sfcf(str(tmp_path) + "/data_o", "test", "f_A", quarks="lquark lquark", wf=0, version="2.0",
+                         files=[["cfg" + str(i) for i in range(1, 11, 2)], ["cfg" + str(i) for i in range(2, 11, 2)], ["cfg" + str(i) for i in range(1, 11, 2)]])
+    print(f_A)
+    assert len(f_A) == 3
+    assert list(f_A[0].shape.keys()) == ["test_|r0", "test_|r1", "test_|r2"]
+    assert f_A[0].value == 65.4711887279723
+    assert f_A[1].value == 1.0447210336915187
+    assert f_A[2].value == -41.025094911185185
+
+
 def test_o_bib(tmp_path):
     build_test_environment(str(tmp_path), "o", 5, 3)
     f_V0 = sfin.read_sfcf(str(tmp_path) + "/data_o", "test", "F_V0", quarks="lquark lquark", wf=0, wf2=0, version="2.0", corr_type="bib")
@@ -120,6 +132,25 @@ def test_c_bi(tmp_path):
     assert f_A[2].value == -41.025094911185185
 
 
+def test_c_bi_files(tmp_path):
+    build_test_environment(str(tmp_path), "c", 10, 3)
+    f_A = sfin.read_sfcf(str(tmp_path) + "/data_c", "data_c", "f_A", quarks="lquark lquark", wf=0, version="2.0c",
+                         files=[["data_c_r0_n" + str(i) for i in range(1, 11, 2)], ["data_c_r1_n" + str(i) for i in range(2, 11, 2)], ["data_c_r2_n" + str(i) for i in range(1, 11, 2)]])
+    print(f_A)
+    assert len(f_A) == 3
+    assert list(f_A[0].shape.keys()) == ["data_c_|r0", "data_c_|r1", "data_c_|r2"]
+    assert f_A[0].value == 65.4711887279723
+    assert f_A[1].value == 1.0447210336915187
+    assert f_A[2].value == -41.025094911185185
+
+
+def test_c_bi_files_int_fail(tmp_path):
+    build_test_environment(str(tmp_path), "c", 10, 3)
+    with pytest.raises(TypeError):
+        sfin.read_sfcf(str(tmp_path) + "/data_c", "data_c", "f_A", quarks="lquark lquark", wf=0, version="2.0c",
+                       files=[[range(1, 11, 2)], [range(2, 11, 2)], [range(1, 11, 2)]])
+
+
 def test_c_bib(tmp_path):
     build_test_environment(str(tmp_path), "c", 5, 3)
     f_V0 = sfin.read_sfcf(str(tmp_path) + "/data_c", "data_c", "F_V0", quarks="lquark lquark", wf=0, wf2=0, version="2.0c", corr_type="bib")
@@ -256,6 +287,24 @@ def test_a_bi(tmp_path):
     assert f_A[2].value == -41.025094911185185
 
 
+def test_a_bi_files(tmp_path):
+    build_test_environment(str(tmp_path), "a", 5, 3)
+    f_A = sfin.read_sfcf(str(tmp_path) + "/data_a", "data_a", "f_A", quarks="lquark lquark", wf=0, version="2.0a", files=["data_a_r0.f_A", "data_a_r1.f_A", "data_a_r2.f_A"])
+    print(f_A)
+    assert len(f_A) == 3
+    assert list(f_A[0].shape.keys()) == ["data_a_|r0", "data_a_|r1", "data_a_|r2"]
+    assert f_A[0].value == 65.4711887279723
+    assert f_A[1].value == 1.0447210336915187
+    assert f_A[2].value == -41.025094911185185
+
+
+def test_a_bi_files_int_fail(tmp_path):
+    build_test_environment(str(tmp_path), "a", 10, 3)
+    with pytest.raises(TypeError):
+        sfin.read_sfcf(str(tmp_path) + "/data_a", "data_a", "f_A", quarks="lquark lquark", wf=0, version="2.0a",
+                       files=[[range(1, 11, 2)], [range(2, 11, 2)], [range(1, 11, 2)]])
+
+
 def test_a_bib(tmp_path):
     build_test_environment(str(tmp_path), "a", 5, 3)
     f_V0 = sfin.read_sfcf(str(tmp_path) + "/data_a", "data_a", "F_V0", quarks="lquark lquark", wf=0, wf2=0, version="2.0a", corr_type="bib")