Repository: mne-tools/mne-python

Issue:
info['meas_date'] changes during I/O roundtrip
#### Describe the bug
Reading some test data, setting its `meas_date` to `None` via `raw.set_meas_date()`, writing the altered file to disk, and reading it back produced an unexpected result: `meas_date` is **not** `None` after reading, although it certainly was before writing.

#### Steps to reproduce

```python
import mne
import pathlib

mne.set_log_level('error')


data_path = pathlib.Path(mne.datasets.testing.data_path())
in_fname = data_path / 'MEG' /'sample' / 'sample_audvis_trunc_raw.fif'
out_fname = '/tmp/foo.fif'
meas_date = dict()

raw = mne.io.read_raw_fif(in_fname)
meas_date['orig'] = raw.info['meas_date']

raw.set_meas_date(None)
meas_date['changed'] = raw.info['meas_date']

raw.save(out_fname, overwrite=True)
raw = mne.io.read_raw_fif(out_fname)
meas_date['read_back'] = raw.info['meas_date']

for k, v in meas_date.items():
    print(f'{k}:\t{v}')
```

#### Expected results
```
orig:	2002-12-03 19:01:10.720100+00:00
changed:	None
read_back:	None
```

#### Actual results
```
orig:	2002-12-03 19:01:10.720100+00:00
changed:	None
read_back:	2014-09-19 17:05:52+00:00
```

cc @agramfort 


Discussion:
meas_date is not one of the info entries that should be manually changed. You need to use the anonymize functions. meas_date is inferred from elsewhere when the file is not fully anonymized.
> meas_date is not one of the info entries that should be manually changed.

Wow, but, but, but… Why do we have a public `set_meas_date()` method then??
> Wow, but, but, but… Why do we have a public `set_meas_date()` method then??

And, I always thought manually changing stuff in `info` was totally fine… After all, that is also how I change `bads`, subject info etc… If I'm only supposed to ever touch a subset of the `info`  values, we should probably rethink access to  this data structure.
> meas_date is inferred from elsewhere when the file is not fully anonymized.

Is this always the case, or does it only apply when `meas_date` is `None`?
Related:
#7772
#7774
For fun weekend reading here is the last time this was discussed:

#7067
#7071
Oh noes. I would like to discuss whether and how we could make `info` read-only and only allow alterations via dedicated setter functions (while keeping the ability to change the values of a few"safe" keys directly, for backward-compat). Anyone willing to chime in? ;)

Because it _is_ really inconsistent and error-provoking. I'm being taught in the tutorials to manually fiddle with `info['bads']`, and now suddenly I'm being told that I actually shouldn't be touching most of the items in the `info` dict. – This is just _in addition_ to _having_ a `set_meas_date` function already, that will apparently not always really do what its name suggests, and leads to a situation where I write data to disk but reading it back doesn't produce the same stuff I had before writing. How is an ordinary user ever going to figure this out?

cc @cbrnr @drammock 
> Oh noes. I would like to discuss whether and how we could make info read-only and only allow alterations via dedicated setter functions (while keeping the ability to change the values of a few"safe" keys directly, for backward-compat). Anyone willing to chime in? ;)

Read this first:

https://github.com/mne-tools/mne-python/issues/2300
https://github.com/mne-tools/mne-python/pull/2765
https://github.com/mne-tools/mne-python/issues/2860
https://github.com/mne-tools/mne-python/pull/3147

I'd be up for reconsidering adding it. I think now that we require an actual `Info` everywhere, it should be possible to at least `warn` people when they change an entry that they shouldn't, and maybe provide some expert or internal interface to allow it. I imagine we could do it with maybe a dozen lines in the `Info` class itself.

But even this wouldn't be the end of the problem. The entries in `info` itself are mutable (dict, list, ndarray). A separable step would be changing as many mutable entries of `Info` (e.g., list) to immutable ones (tuple), maybe also using some immutable dict where possible (e.g., `info['chs'][ii]`). FWIW this is also an issue when deep-copying info that we currently use some specialized code to deal with, so in the long run this step at least might make some things simpler.

> Because it is really inconsistent and error-provoking. I'm being taught in the tutorials to manually fiddle with info['bads'], and now suddenly I'm being told that I actually shouldn't be touching most of the items in the info dict.

To start, we should update the early tutorial docs (if we haven't already) to make it clearer. Or make it clearer (with a `.. warning::` perhaps?) in the `Info` [docstring](https://mne.tools/dev/generated/mne.Info.html) that it says:

    The only entries that should be manually changed by the user are info['bads']
    and info['description']. All other entries should be considered read-only,
    though they can be modified by various MNE-Python functions or methods
    (which have safeguards to ensure all fields remain in sync).

> – This is just in addition to having a set_meas_date function already, that will apparently not always really do what its name suggests, and leads to a situation where I write data to disk but reading it back doesn't produce the same stuff I had before writing.

This is a bug with `set_meas_date`. I thought you were doing `info['meas_date'] = None` manually, which is a no-no -- `set_meas_date` should indeed be used.
I agree that the `info` dict is not entirely ideal. I think this is due to two reasons: (1) it has grown historically/organically, and (2) it is reflecting the internal FIF data structure. Therefore, it's not easy to change this structure, but at least such things should be clearly documented.

In fact, the [documentation of `mne.Info`](https://mne.tools/dev/generated/mne.Info.html) says:

> "The only entries that should be manually changed by the user are info['bads'] and info['description']. All other entries should be considered read-only, though they can be modified by various MNE-Python functions or methods (which have safeguards to ensure all fields remain in sync).

The [`mne.Info` tutorial](https://mne.tools/dev/auto_tutorials/intro/plot_30_info.html#sphx-glr-auto-tutorials-intro-plot-30-info-py) doesn't really discuss how to write the various fields, so maybe we should add a dedicated section about this topic.

Regarding setters, I'm not a huge fan. The more Pythonic way is to use properties, which is really something I'd like to see more. I know the argument setters vs. properties (properties shouldn't be long-running functions), but most setters here are more or less simple assignments with some sanity checks, so most of the setters could actually be implemented as properties. For users, this makes everything much easier to figure out, because everything is changed directly via assignment (and only internally properties do the work under the hood).

> I'd be up for reconsidering adding it. I think now that we require an actual Info everywhere, it should be possible to at least warn people when they change an entry that they shouldn't, and maybe provide some expert or internal interface to allow it. I imagine we could do it with maybe a dozen lines in the Info class itself.

> But even this wouldn't be the end of the problem. The entries in info itself are mutable (dict, list, ndarray). A separable step would be changing as many mutable entries of Info (e.g., list) to immutable ones (tuple), maybe also using some immutable dict where possible (e.g., info['chs'][ii]). FWIW this is also an issue when deep-copying info that we currently use some specialized code to deal with, so in the long run this step at least might make some things simpler.

This is over-engineering IMO. Properties would make things much easier for users.

> This is a bug with set_meas_date. I thought you were doing info['meas_date'] = None manually, which is a no-no -- set_meas_date should indeed be used.

Again, `info['meas_date'] = None` is much more intuitive and Pythonic. Let's implement properties.
> Regarding setters, I'm not a huge fan. The more Pythonic way is to use properties, which is really something I'd like to see more. I know the argument setters vs. properties (properties shouldn't be long-running functions), but most setters here are more or less simple assignments with some sanity checks, so most of the setters could actually be implemented as properties.

This to me seems like a mostly orthogonal discussion -- here we are mostly talking about which properties should be settable at all and how to prevent errors. You are talking about how they should be set (and requiring deprecations of existing functions, etc.). Feel free to open a separate issue for this but I'd rather keep the discussion here focused on the read-only-ness / documentation of `Info`.
@cbrnr 
> Properties would make things much easier for users.

Yes I agree that properties are the more Pythonic approach and should be preferred.
> This to me seems like a mostly orthogonal discussion -- here we are mostly talking about which properties should be settable at all and how to prevent errors. You are talking about how they should be set (and requiring deprecations of existing functions, etc.). Feel free to open a separate issue for this but I'd rather keep the discussion here focused on the read-only-ness / documentation of Info.

True, but properties provide a Pythonic way to implement read-only fields - I'll open a separate issue to discuss.

I agree that here the best solution is to improve the docs (e.g. make the existing note more obvious as you suggested).
Just to get back to the original issue:

> > – This is just in addition to having a set_meas_date function already, that will apparently not always really do what its name suggests, and leads to a situation where I write data to disk but reading it back doesn't produce the same stuff I had before writing.
> 
> This is a bug with `set_meas_date`. I thought you were doing `info['meas_date'] = None` manually

… it's this one, in case anyone was wondering after our digression here. 😅
yes please focus what is the matter here. @hoechenberg uses a public method
set_meas_date
passes None, gets no warning as everything seems fine and then he hits
a legitimate surprise on read. So either we add a warning that says that
None
should not be passed or we consider this a bug and we fix it. I was
surprised
myself by the failure of the IO round trip and I suggested to open the
issue.
Adding code with eg properties is add a lot of complexity for an issue that
is for me orthogonal and info['meas_date'] = None  was not done

>


Resolving patch:
diff --git a/mne/channels/channels.py b/mne/channels/channels.py
--- a/mne/channels/channels.py
+++ b/mne/channels/channels.py
@@ -24,6 +24,7 @@
                        _check_excludes_includes, _contains_ch_type,
                        channel_indices_by_type, pick_channels, _picks_to_idx,
                        _get_channel_types)
+from ..io.write import DATE_NONE
 
 
 def _get_meg_system(info):
@@ -608,6 +609,21 @@ def set_meas_date(self, meas_date):
         from ..annotations import _handle_meas_date
         meas_date = _handle_meas_date(meas_date)
         self.info['meas_date'] = meas_date
+
+        # clear file_id and meas_id if needed
+        if meas_date is None:
+            for key in ('file_id', 'meas_id'):
+                value = self.info.get(key)
+                if value is not None:
+                    assert 'msecs' not in value
+                    value['secs'] = DATE_NONE[0]
+                    value['usecs'] = DATE_NONE[1]
+                    # The following copy is needed for a test CTF dataset
+                    # otherwise value['machid'][:] = 0 would suffice
+                    _tmp = value['machid'].copy()
+                    _tmp[:] = 0
+                    value['machid'] = _tmp
+
         if hasattr(self, 'annotations'):
             self.annotations._orig_time = meas_date
         return self


Test patch:
diff --git a/mne/io/fiff/tests/test_raw_fiff.py b/mne/io/fiff/tests/test_raw_fiff.py
--- a/mne/io/fiff/tests/test_raw_fiff.py
+++ b/mne/io/fiff/tests/test_raw_fiff.py
@@ -1274,6 +1274,12 @@ def test_save(tmpdir):
     assert_array_equal(annot.description, new_raw.annotations.description)
     assert annot.orig_time == new_raw.annotations.orig_time
 
+    # test set_meas_date(None)
+    raw.set_meas_date(None)
+    raw.save(new_fname, overwrite=True)
+    new_raw = read_raw_fif(new_fname, preload=False)
+    assert new_raw.info['meas_date'] is None
+
 
 @testing.requires_testing_data
 def test_annotation_crop(tmpdir):