Repository: mne-tools/mne-bids

Issue:
An updated function in the API for updating scans.tsv and participants files
#### Describe the problem
When you're writing datasets using the `write_raw_bids` functions, sometimes you might manually delete a participant, or some scans. However, this needs to get reflected in the corresponding files. The only way to do that is manually right now.

#### Describe your solution
A `update_scans_tsv()` and `update_participants_files` function perhaps to do that book-keeping for you. I'm not sure what the best looking API would look like, but if people agree, then I can put together something and a short "example?

The U in CRUD:

```Python

def update_scans_sidecar(bids_root, subject=None, session=None):
     # if a specific subject/session, then only do that specific scans.tsv
     # else loop through all files
 
     # load scans tsv file
     # delete row if that file doesn't exist anymore
```


```Python

def update_participants_sidecar(bids_root):
     # get all subject ids inside bids_root
 
     # load participants tsv file
     # delete row if that subject doesn't exist anymore
```

#### Describe possible alternatives
Possibly the function could be combined into one? Allowing one to update sidecar files in general? Idk how that would look though because we definitely can't account for all sidecars.

#### Additional context
Minimally keeping the scans and participants file up to date would be nice.



Discussion:
it sounds like a reasonable request to me

@hoechenberger @sappelhoff wdyt?
This is something that has been on our "want-to-have" list for a while now, so: yes, I want to have this ;) I will need to think about which solution / API would make most sense, though. I believe we should try to sketch this out a little before you start with an implementation, @adam2392.

@agramfort may also have some ideas on this.
what we need is a basic CRUD (in DB sense) for certain files. Participants,
bad channels, coregistration info

>

yes, I agree it's important to replace this "manual fiddling around" with code as far as possible :+1: 
> This is something that has been on our "want-to-have" list for a while now, so: yes, I want to have this ;) I will need to think about which solution / API would make most sense, though. I believe we should try to sketch this out a little before you start with an implementation, @adam2392.

What do you think of the proposed two functions, I put into the issue just now?


would be the signature of these two functions? Can you share a snippet
to illustrate the use case?

>

Hey @adam2392, sorry I haven't gotten back to you yet, I was busy with other stuff. Like @agramfort said, an API proposal could be very helpful to get a discussion started! :)

Resolving patch:
diff --git a/mne_bids/path.py b/mne_bids/path.py
--- a/mne_bids/path.py
+++ b/mne_bids/path.py
@@ -37,6 +37,7 @@
     _ensure_tuple,
     warn,
 )
+from mne_bids.tsv_handler import _from_tsv, _drop, _to_tsv
 
 
 def _find_empty_room_candidates(bids_path):
@@ -626,6 +627,172 @@ def mkdir(self, exist_ok=True):
         self.directory.mkdir(parents=True, exist_ok=exist_ok)
         return self
 
+    @verbose
+    def rm(self, *, safe_remove=True, verbose=None):
+        """Safely delete a set of files from a BIDS dataset.
+
+        Deleting a scan that conforms to the bids-validator will
+        remove the respective row in ``*_scans.tsv``,  the
+        corresponding sidecar files, and the data file itself.
+
+        Deleting all files of a subject will update the
+        ``*_participants.tsv`` file.
+
+
+        Parameters
+        ----------
+        safe_remove : bool
+            If ``False``, directly delete and update the files.
+            Otherwise, displays the list of operations planned
+            and asks for user confirmation before
+            executing them (default).
+        %(verbose)s
+
+        Returns
+        -------
+        self : BIDSPath
+            The BIDSPath object.
+
+        Examples
+        --------
+        Remove one specific run:
+
+        >>> bids_path = BIDSPath(subject='01', session='01', run="01",  # doctest: +SKIP
+        ...                      root='/bids_dataset').rm()  # doctest: +SKIP
+        Please, confirm you want to execute the following operations:
+        Delete:
+        /bids_dataset/sub-01/ses-01/meg/sub-01_ses-01_run-01_channels.tsv
+        /bids_dataset/sub-01/ses-01/meg/sub-01_ses-01_run-01_events.json
+        /bids_dataset/sub-01/ses-01/meg/sub-01_ses-01_run-01_events.tsv
+        /bids_dataset/sub-01/ses-01/meg/sub-01_ses-01_run-01_meg.fif
+        /bids_dataset/sub-01/ses-01/meg/sub-01_ses-01_run-01_meg.json
+        Update:
+        /bids_dataset/sub-01/ses-01/sub-01_ses-01_scans.tsv
+        I confirm [y/N]>? y
+
+        Remove all the files of a specific subject:
+
+        >>> bids_path = BIDSPath(subject='01', root='/bids_dataset',  # doctest: +SKIP
+        ...                      check=False).rm()  # doctest: +SKIP
+        Please, confirm you want to execute the following operations:
+        Delete:
+        /bids_dataset/sub-01/ses-01/meg/sub-01_ses-01_acq-calibration_meg.dat
+        /bids_dataset/sub-01/ses-01/meg/sub-01_ses-01_acq-crosstalk_meg.fif
+        /bids_dataset/sub-01/ses-01/meg/sub-01_ses-01_coordsystem.json
+        /bids_dataset/sub-01/ses-01/meg/sub-01_ses-01_run-02_channels.tsv
+        /bids_dataset/sub-01/ses-01/meg/sub-01_ses-01_run-02_events.json
+        /bids_dataset/sub-01/ses-01/meg/sub-01_ses-01_run-02_events.tsv
+        /bids_dataset/sub-01/ses-01/meg/sub-01_ses-01_run-02_meg.fif
+        /bids_dataset/sub-01/ses-01/meg/sub-01_ses-01_run-02_meg.json
+        /bids_dataset/sub-01/ses-01/sub-01_ses-01_scans.tsv
+        /bids_dataset/sub-01
+        Update:
+        /bids_dataset/participants.tsv
+        I confirm [y/N]>? y
+        """
+        # only proceed if root is defined
+        if self.root is None:
+            raise RuntimeError("The root must not be None to remove files.")
+
+        # Planning:
+        paths_matched = self.match(ignore_json=False, check=self.check)
+        subjects = set()
+        paths_to_delete = list()
+        paths_to_update = {}
+        subjects_paths_to_delete = []
+        participants_tsv_fpath = None
+        for bids_path in paths_matched:
+            paths_to_delete.append(bids_path)
+            # if a datatype is present, then check
+            # if a scan is deleted or not
+            if bids_path.datatype is not None:
+                # read in the corresponding scans file
+                scans_fpath = (
+                    bids_path.copy()
+                    .update(datatype=None)
+                    .find_matching_sidecar(
+                        suffix="scans",
+                        extension=".tsv",
+                        on_error="raise",
+                    )
+                )
+                paths_to_update.setdefault(scans_fpath, []).append(bids_path)
+            subjects.add(bids_path.subject)
+
+        files_to_delete = set(p.fpath for p in paths_to_delete)
+        for subject in subjects:
+            # check existence of files in the subject dir
+            subj_path = BIDSPath(root=self.root, subject=subject)
+            subj_files = [
+                fpath for fpath in subj_path.directory.rglob("*") if fpath.is_file()
+            ]
+            if set(subj_files) <= files_to_delete:
+                subjects_paths_to_delete.append(subj_path)
+                participants_tsv_fpath = self.root / "participants.tsv"
+
+        # Informing:
+        pretty_delete_paths = "\n".join(
+            [
+                str(p)
+                for p in paths_to_delete
+                + [p.directory for p in subjects_paths_to_delete]
+            ]
+        )
+        pretty_update_paths = "\n".join(
+            [
+                str(p)
+                for p in list(paths_to_update.keys())
+                + (
+                    [participants_tsv_fpath]
+                    if participants_tsv_fpath is not None
+                    else []
+                )
+            ]
+        )
+        summary = ""
+        if pretty_delete_paths:
+            summary += f"Delete:\n{pretty_delete_paths}\n"
+        if pretty_update_paths:
+            summary += f"Update:\n{pretty_update_paths}\n"
+
+        if safe_remove:
+            choice = input(
+                "Please, confirm you want to execute the following operations:\n"
+                f"{summary}\nI confirm [y/N]"
+            )
+            if choice.lower() != "y":
+                return
+        else:
+            logger.info(f"Executing the following operations:\n{summary}")
+
+        # Execution:
+        for bids_path in paths_to_delete:
+            bids_path.fpath.unlink()
+
+        for scans_fpath, bids_paths in paths_to_update.items():
+            if not scans_fpath.exists():
+                continue
+            # get the relative datatype of these bids files
+            bids_fnames = [op.join(p.datatype, p.fpath.name) for p in bids_paths]
+
+            scans_tsv = _from_tsv(scans_fpath)
+            scans_tsv = _drop(scans_tsv, bids_fnames, "filename")
+            _to_tsv(scans_tsv, scans_fpath)
+
+        subjects_to_delete = []
+        for subj_path in subjects_paths_to_delete:
+            if subj_path.directory.exists():
+                sh.rmtree(subj_path.directory)
+            subjects_to_delete.append(subj_path.subject)
+        if subjects_to_delete and participants_tsv_fpath.exists():
+            participants_tsv = _from_tsv(participants_tsv_fpath)
+            participants_tsv = _drop(
+                participants_tsv, subjects_to_delete, "participant_id"
+            )
+            _to_tsv(participants_tsv, participants_tsv_fpath)
+
+        return self
+
     @property
     def fpath(self):
         """Full filepath for this BIDS file.


Test patch:
diff --git a/mne_bids/tests/test_path.py b/mne_bids/tests/test_path.py
--- a/mne_bids/tests/test_path.py
+++ b/mne_bids/tests/test_path.py
@@ -39,7 +39,6 @@
 
 from test_read import _read_raw_fif, warning_str
 
-
 subject_id = "01"
 session_id = "01"
 run = "01"
@@ -259,6 +258,72 @@ def test_make_folders(tmp_path):
     os.chdir(curr_dir)
 
 
+def test_rm(return_bids_test_dir, capsys, tmp_path_factory):
+    """Test the BIDSPath.rm method to remove files"""
+    # for some reason, mne's logger can't be captured by caplog....
+    bids_root = str(tmp_path_factory.mktemp("test_rm") / "mnebids_utils_test_bids_ds")
+    shutil.copytree(return_bids_test_dir, bids_root)
+
+    # without providing all the entities, ambiguous when trying
+    # to use fpath
+    bids_path = BIDSPath(
+        subject=subject_id,
+        session=session_id,
+        run="01",
+        acquisition=acq,
+        task=task,
+        root=bids_root,
+    )
+
+    # Delete one run:
+    deleted_paths = bids_path.match(ignore_json=False)
+    updated_paths = [
+        bids_path.copy()
+        .update(datatype=None)
+        .find_matching_sidecar(
+            suffix="scans",
+            extension=".tsv",
+            on_error="raise",
+        )
+    ]
+    expected = ["Executing the following operations:", "Delete:", "Update:", ""]
+    expected += [str(p) for p in deleted_paths + updated_paths]
+    bids_path.rm(safe_remove=False, verbose="INFO")
+    captured = capsys.readouterr().out
+    assert set(captured.splitlines()) == set(expected)
+
+    # delete the last run of a subject:
+    bids_path = BIDSPath(
+        subject=subject_id,
+        session=session_id,
+        root=bids_root,
+    )
+    deleted_paths = bids_path.match(ignore_json=False)
+    deleted_paths += [
+        BIDSPath(
+            root=bids_path.root,
+            subject=bids_path.subject,
+        ).directory
+    ]
+    updated_paths = [
+        bids_path.copy()
+        .update(datatype=None)
+        .find_matching_sidecar(
+            suffix="scans",
+            extension=".tsv",
+            on_error="raise",
+        ),
+        bids_path.root / "participants.tsv",
+    ]
+    expected = ["Executing the following operations:", "Delete:", "Update:", ""]
+    expected += [str(p) for p in deleted_paths + updated_paths]
+    bids_path.rm(safe_remove=False, verbose="INFO")
+    captured2 = capsys.readouterr().out
+    assert set(captured2.splitlines()) == set(expected)
+    print("\n".join(captured))
+    print("\n".join(captured2))
+
+
 def test_parse_ext():
     """Test the file extension extraction."""
     f = "sub-05_task-matchingpennies.vhdr"