Repository: mne-tools/mne-python

Issue:
2GB .fif limit
Hi everyone,

Mid-term MNE-Python user here. I have never run into saving FIF files before, because if I have a dataset with around 6GB worth of data, whenever I save my epochs, MNE-Python usually parcellates them into many < 2GB files and appends "-1", "-2" onto the filenames. Then, all I have to do is load the main file and it automatically detects that a split has happened and iteratively loads all these < 2GB divisions of the FIF data and puts it all back together again into the varriable assigned to `mne.read_epochs`.

Now as I am dealing with more and more data, I notice a few edge cases. It seems sometimes that I get the error telling me I can't save a file due to this < 2GB restriction, when this is much smaller than another file that has saved successfully. 

My analysis pipeline requires larger subsets and smaller subsets taken from the same data. The large data saves fine (MNE-Python splits the files equally) but I am noticing now that in many of the cases where I extract subsets, the splitting fails and it tries to create > 2GB files in my current directory and then calls the error.

I am not sure how it works but it looks like there is a calculation of sorts that determines what to do when the data is large. The subsets could have easily been split up multiple < 2GB files, but this isn't happening if the data is, say, slightly over the limit.

I can describe the saving process as having three main cases:

* Data is < 2GB, `epochs.save` works fine
* Data is >>>> 2GB (way over), `epochs.save` splits the files and successfully saves
* Data is "slightly" over 2GB, `epochs.save` doesn't split, calls exception

Does anyone have any insight into the exact condition checking and can see if there is a bug somewhere? It's awkward to keep splitting the data for saving and concatenating them back together when I need them. Those processes take a fair bit of time and ideally I would like not to have to split them arbitrarily when they're slightly over the limit.

If it's not a bug, then that's fine. I am just interested in knowing why it works that way.
So, if anyone can shed some light, it's be very much appreciated.

I was trying to think of a code snippet to demonstrate the example, but that's problematic given it's a large-data problem and not something easily demonstrated with a nice, clean, reproducible example (sorry!)

Best
Alex


Discussion:
Should be fixed in master by #7740. Can you try it on one of your problematic cases? We didn't backport to 0.20 but maybe we should.
I've now backported #7740 to maint/0.20, @agramfort worth another quick release for this?
sure why not.

>

Thanks, guys! I will check it out and get back if any further issues.
Keep up the good work!



Hi everyone,

This problem has reared its head again. I had it originally in version 0.20.6 and then updated just now to 0.21.0 and I'm still getting the error. Basically, able to save a 13 GB Epochs instance and it splits it over 3-4 FIFF files and numbers the parts to it. I then equally balance all of the data in the respective classes, concatenate the epochs into a new data structure and try to save and I get the following error:

`OSError: FIFF file exceeded 2GB limit, please split file or save to a different format`

The new file is only 9.24 GB and MNE had no trouble saving the 13 GB version, but it doesn't like my splitting, extracting and concatenation of the parts. I had this issue earlier and an update was released and this fixed the problem, at least I thought. So, I'm not sure how connected the fix is (but the error is identical).


Here is the full trace if it helps:


```
OSError                                   Traceback (most recent call last)
<ipython-input-18-74621510a84e> in <module>
      1 # save
----> 2 epochs_concat_reduced.save('bigram_10class_equal_epochs-epo.fif', overwrite=True)

<decorator-gen-199> in save(self, fname, split_size, fmt, overwrite, verbose)

~\AppData\Local\Continuum\anaconda3\lib\site-packages\mne\epochs.py in save(self, fname, split_size, fmt, overwrite, verbose)
   1647             # avoid missing event_ids in splits
   1648             this_epochs.event_id = self.event_id
-> 1649             _save_split(this_epochs, fname, part_idx, n_parts, fmt)
   1650 
   1651     def equalize_event_counts(self, event_ids, method='mintime'):

~\AppData\Local\Continuum\anaconda3\lib\site-packages\mne\epochs.py in _save_split(epochs, fname, part_idx, n_parts, fmt)
    185     end_block(fid, FIFF.FIFFB_PROCESSED_DATA)
    186     end_block(fid, FIFF.FIFFB_MEAS)
--> 187     end_file(fid)
    188 
    189 

~\AppData\Local\Continuum\anaconda3\lib\site-packages\mne\io\write.py in end_file(fid)
    335     """Write the closing tags to a fif file and closes the file."""
    336     write_nop(fid, last=True)
--> 337     check_fiff_length(fid)
    338     fid.close()
    339 

~\AppData\Local\Continuum\anaconda3\lib\site-packages\mne\io\write.py in check_fiff_length(fid, close)
    328         if close:
    329             fid.close()
--> 330         raise IOError('FIFF file exceeded 2GB limit, please split file or '
    331                       'save to a different format')
    332 

```
@Alex159 can you try replicating this problem using `mne.EpochsArray(np.zeros(...), create_info(...)).save(...)`? Presumably if you use the same dimensions as the data you're trying to save you'll hit the problem.

If it doesn't work with EpochsArray it suggests that there is something about your Epochs that we aren't accounting for when we try to figure out where to split things.
@larsoner Hi Eric, thanks for the response. I can confirm that using the code in your response, everything saves fine. So, it looks like there is something wrong with the EpochsArray. I think I might have worked it out though. I am concatenating a lot of events and it looks like I hit numeric overflow in the first column of the events array. Everything is increasing right until the end and then I have these extreme negative numbered event time samples alongside a numeric overflow. That then leads to a 'samples not in chronological order' warning - and that must be what leads to the inability to save.

I will take out the epochs from the concatenation operation such that no numerical overflow happens and will try to save the data again. I suspect this will work and the overflow / lack of chronological ordering is the issue at hand.

----

Yes, it works fine now. The issue relates to asynchronous events brought about by having event timings over the maximum limit of an int32 number (> 2,147,483,647).

Can I just change the dtype of the events array to be of int64 instead of int32 without causing any problems, do you think?
I'm guessing I'd need to change it for all of the EpochsArray items that are being concatenated, such that `mne.concatenate_epochs` retains the same dtype in its events array?

are you on windows? cause windows used int32 by default for integers

>

We should make sure we use `np.int64` explicitly for the `events` array in our code. @Alex159 can you track down whether the problem arose from you creating `events` using `int` on windows (which will be 32 bit as @agramfort says) or if it's a problem with our code using `int` somewhere? It could even need fixing both places...
Actually it looks like we `write_int` the events, which for FIFF means they are 32-bit integers. Given that there are no int64 in FIFF, I think we should probably check that the events are all <= 2147483647. @Alex159 I think in your case you will need to use smaller values for the `events` array :( You could add the larger numbers as `metadata` perhaps
Yes, on Windows for this analysis as data is restricted to the university network. I can run something similar on a Linux cluster and see the results, but will have to wait until tomorrow to try that part. I managed to make `concatenate_epochs` use int64 dtype by changing all the dtypes of the EpochArrays that are being concatenated. This did avoid the overflow in the event time samples, meaning they now do stay in chronological order. However, another overflow messages arises somewhere I can't quite locate and the issue with saving becomes a problem again.

I took out the number of epochs to keep the concatenated events within the int32 limit and managed to successfully save, where MNE correctly splits it all up evenly. So the issue seems to be connected to the int32 limit, but is not specifically related to event time samples being in ascending order. I don't know enough about the internals to figure out what the issue could be.

It seems as if @larsoner has pointed out that the FIFF code does imply some sort of restriction here. The Linux test will be done tomorrow.

Responding to the question about using smaller values for the `events` array, how can I go about that?
Is it possible to just overwrite the event times without destroying the ability to pull out the correct data when requested? 
If that's possible, seems like a good interim solution!

Resolving patch:
diff --git a/mne/epochs.py b/mne/epochs.py
--- a/mne/epochs.py
+++ b/mne/epochs.py
@@ -577,6 +577,19 @@ def _check_consistency(self):
         assert all(isinstance(log, tuple) for log in self.drop_log)
         assert all(isinstance(s, str) for log in self.drop_log for s in log)
 
+    def reset_drop_log_selection(self):
+        """Reset the drop_log and selection entries.
+
+        This method will simplify ``self.drop_log`` and ``self.selection``
+        so that they are meaningless (tuple of empty tuples and increasing
+        integers, respectively). This can be useful when concatenating
+        many Epochs instances, as ``drop_log`` can accumulate many entries
+        which can become problematic when saving.
+        """
+        self.selection = np.arange(len(self.events))
+        self.drop_log = (tuple(),) * len(self.events)
+        self._check_consistency()
+
     def load_data(self):
         """Load the data if not already preloaded.
 
@@ -1614,13 +1627,15 @@ def save(self, fname, split_size='2GB', fmt='single', overwrite=False,
         # check for file existence
         _check_fname(fname, overwrite)
 
-        split_size = _get_split_size(split_size)
+        split_size_bytes = _get_split_size(split_size)
 
         _check_option('fmt', fmt, ['single', 'double'])
 
         # to know the length accurately. The get_data() call would drop
         # bad epochs anyway
         self.drop_bad()
+        # total_size tracks sizes that get split
+        # over_size tracks overhead (tags, things that get written to each)
         if len(self) == 0:
             warn('Saving epochs with no data')
             total_size = 0
@@ -1630,37 +1645,69 @@ def save(self, fname, split_size='2GB', fmt='single', overwrite=False,
             assert d.dtype in ('>f8', '<f8', '>c16', '<c16')
             total_size = d.nbytes * len(self)
         self._check_consistency()
+        over_size = 0
         if fmt == "single":
             total_size //= 2  # 64bit data converted to 32bit before writing.
-        total_size += 32  # FIF tags
+        over_size += 32  # FIF tags
         # Account for all the other things we write, too
         # 1. meas_id block plus main epochs block
-        total_size += 132
+        over_size += 132
         # 2. measurement info (likely slight overestimate, but okay)
-        total_size += object_size(self.info) + 16 * len(self.info)
+        over_size += object_size(self.info) + 16 * len(self.info)
         # 3. events and event_id in its own block
-        total_size += (self.events.size * 4 +
-                       len(_event_id_string(self.event_id)) + 72)
+        total_size += self.events.size * 4
+        over_size += len(_event_id_string(self.event_id)) + 72
         # 4. Metadata in a block of its own
         if self.metadata is not None:
-            total_size += len(_prepare_write_metadata(self.metadata)) + 56
+            total_size += len(_prepare_write_metadata(self.metadata))
+        over_size += 56
         # 5. first sample, last sample, baseline
-        total_size += 40 + 40 * (self.baseline is not None)
-        # 6. drop log
-        total_size += len(json.dumps(self.drop_log)) + 16
+        over_size += 40 * (self.baseline is not None) + 40
+        # 6. drop log: gets written to each, with IGNORE for ones that are
+        #    not part of it. So make a fake one with all having entries.
+        drop_size = len(json.dumps(self.drop_log)) + 16
+        drop_size += 8 * (len(self.selection) - 1)  # worst case: all but one
+        over_size += drop_size
         # 7. reject params
         reject_params = _pack_reject_params(self)
         if reject_params:
-            total_size += len(json.dumps(reject_params)) + 16
+            over_size += len(json.dumps(reject_params)) + 16
         # 8. selection
-        total_size += self.selection.size * 4 + 16
+        total_size += self.selection.size * 4
+        over_size += 16
         # 9. end of file tags
-        total_size += _NEXT_FILE_BUFFER
+        over_size += _NEXT_FILE_BUFFER
+        logger.debug(f'    Overhead size:   {str(over_size).rjust(15)}')
+        logger.debug(f'    Splittable size: {str(total_size).rjust(15)}')
+        logger.debug(f'    Split size:      {str(split_size_bytes).rjust(15)}')
+        # need at least one per
+        n_epochs = len(self)
+        n_per = total_size // n_epochs if n_epochs else 0
+        min_size = n_per + over_size
+        if split_size_bytes < min_size:
+            raise ValueError(
+                f'The split size {split_size} is too small to safely write '
+                'the epochs contents, minimum split size is '
+                f'{sizeof_fmt(min_size)} ({min_size} bytes)')
 
         # This is like max(int(ceil(total_size / split_size)), 1) but cleaner
-        n_parts = (total_size - 1) // split_size + 1
-        assert n_parts >= 1
-        epoch_idxs = np.array_split(np.arange(len(self)), n_parts)
+        n_parts = max(
+            (total_size - 1) // (split_size_bytes - over_size) + 1, 1)
+        assert n_parts >= 1, n_parts
+        if n_parts > 1:
+            logger.info(f'Splitting into {n_parts} parts')
+            if n_parts > 100:  # This must be an error
+                raise ValueError(
+                    f'Split size {split_size} would result in writing '
+                    f'{n_parts} files')
+
+        if len(self.drop_log) > 100000:
+            warn(f'epochs.drop_log contains {len(self.drop_log)} entries '
+                 f'which will incur up to a {sizeof_fmt(drop_size)} writing '
+                 f'overhead (per split file), consider using '
+                 'epochs.reset_drop_log_selection() prior to writing')
+
+        epoch_idxs = np.array_split(np.arange(n_epochs), n_parts)
 
         for part_idx, epoch_idx in enumerate(epoch_idxs):
             this_epochs = self[epoch_idx] if n_parts > 1 else self
@@ -2521,7 +2568,9 @@ def _read_one_epoch_file(f, tree, preload):
                 selection = np.array(tag.data)
             elif kind == FIFF.FIFF_MNE_EPOCHS_DROP_LOG:
                 tag = read_tag(fid, pos)
-                drop_log = tuple(tuple(x) for x in json.loads(tag.data))
+                drop_log = tag.data
+                drop_log = json.loads(drop_log)
+                drop_log = tuple(tuple(x) for x in drop_log)
             elif kind == FIFF.FIFF_MNE_EPOCHS_REJECT_FLAT:
                 tag = read_tag(fid, pos)
                 reject_params = json.loads(tag.data)


Test patch:
diff --git a/mne/tests/test_epochs.py b/mne/tests/test_epochs.py
--- a/mne/tests/test_epochs.py
+++ b/mne/tests/test_epochs.py
@@ -1051,10 +1051,7 @@ def test_epochs_io_preload(tmpdir, preload):
     epochs = Epochs(raw, events, dict(foo=1, bar=999), tmin, tmax,
                     picks=picks, on_missing='ignore')
     epochs.save(temp_fname, overwrite=True)
-    split_fname_1 = temp_fname[:-4] + '-1.fif'
-    split_fname_2 = temp_fname[:-4] + '-2.fif'
-    assert op.isfile(temp_fname)
-    assert not op.isfile(split_fname_1)
+    _assert_splits(temp_fname, 0, np.inf)
     epochs_read = read_epochs(temp_fname, preload=preload)
     assert_allclose(epochs.get_data(), epochs_read.get_data(), **tols)
     assert_array_equal(epochs.events, epochs_read.events)
@@ -1070,16 +1067,10 @@ def test_epochs_io_preload(tmpdir, preload):
     epochs.save(temp_fname, split_size=split_size, overwrite=True)
     # ... but we correctly account for the other stuff we need to write,
     # so end up with two files ...
-    assert op.isfile(temp_fname)
-    assert op.isfile(split_fname_1)
-    assert not op.isfile(split_fname_2)
+    _assert_splits(temp_fname, 1, split_size_bytes)
     epochs_read = read_epochs(temp_fname, preload=preload)
     # ... and none of the files exceed our limit.
-    for fname in (temp_fname, split_fname_1):
-        with open(fname, 'r') as fid:
-            fid.seek(0, 2)
-            fsize = fid.tell()
-        assert fsize <= split_size_bytes
+    _assert_splits(temp_fname, 1, split_size_bytes)
     assert_allclose(epochs.get_data(), epochs_read.get_data(), **tols)
     assert_array_equal(epochs.events, epochs_read.events)
     assert_array_equal(epochs.selection, epochs_read.selection)
@@ -1099,8 +1090,8 @@ def test_epochs_io_preload(tmpdir, preload):
 
 
 @pytest.mark.parametrize('split_size, n_epochs, n_files, size', [
-    ('1MB', 9, 3, 1024 * 1024),
-    # ('2MB', 18, 2, 2 * 1024 * 1024),  # Useful for debugging
+    ('1.5MB', 9, 6, 1572864),
+    ('3MB', 18, 3, 3 * 1024 * 1024),
 ])
 @pytest.mark.parametrize('metadata', [
     False,
@@ -1137,14 +1128,7 @@ def test_split_saving(tmpdir, split_size, n_epochs, n_files, size, metadata,
     epochs.save(fname, split_size=split_size, overwrite=True)
     got_size = _get_split_size(split_size)
     assert got_size == size
-    written_fnames = [fname] + [
-        fname[:-4] + '-%d.fif' % ii for ii in range(1, n_files + 1)]
-    for this_fname in written_fnames:
-        assert op.isfile(this_fname)
-        with open(this_fname, 'r') as fid:
-            fid.seek(0, 2)
-            file_size = fid.tell()
-        assert size * 0.5 < file_size <= size
+    _assert_splits(fname, n_files, size)
     assert not op.isfile(f'{fname[:-4]}-{n_files + 1}.fif')
     for preload in (True, False):
         epochs2 = mne.read_epochs(fname, preload=preload)
@@ -1152,6 +1136,53 @@ def test_split_saving(tmpdir, split_size, n_epochs, n_files, size, metadata,
         assert_array_equal(epochs.events, epochs2.events)
 
 
+def test_split_many_reset(tmpdir):
+    """Test splitting with many events and using reset."""
+    data = np.zeros((1000, 1, 1024))  # 1 ch, 1024 samples
+    assert data[0, 0].nbytes == 8192  # 8 kB per epoch
+    info = mne.create_info(1, 1000., 'eeg')
+    selection = np.arange(len(data)) + 100000
+    epochs = EpochsArray(data, info, tmin=0., selection=selection)
+    assert len(epochs.drop_log) == 101000
+    assert len(epochs) == len(data) == len(epochs.events)
+    fname = str(tmpdir.join('temp-epo.fif'))
+    for split_size in ('0.5MB', '1MB', '2MB'):  # tons of overhead from sel
+        with pytest.raises(ValueError, match='too small to safely'):
+            epochs.save(fname, split_size=split_size, verbose='debug')
+    with pytest.raises(ValueError, match='would result in writing'):  # ~200
+        epochs.save(fname, split_size='2.27MB', verbose='debug')
+    with pytest.warns(RuntimeWarning, match='writing overhead'):
+        epochs.save(fname, split_size='3MB', verbose='debug')
+    epochs_read = read_epochs(fname)
+    assert_allclose(epochs.get_data(), epochs_read.get_data())
+    assert epochs.drop_log == epochs_read.drop_log
+    mb = 3 * 1024 * 1024
+    _assert_splits(fname, 6, mb)
+    # reset, then it should work
+    fname = str(tmpdir.join('temp-reset-epo.fif'))
+    epochs.reset_drop_log_selection()
+    epochs.save(fname, split_size=split_size, verbose='debug')
+    _assert_splits(fname, 4, mb)
+    epochs_read = read_epochs(fname)
+    assert_allclose(epochs.get_data(), epochs_read.get_data())
+
+
+def _assert_splits(fname, n, size):
+    __tracebackhide__ = True
+    assert n >= 0
+    next_fnames = [fname] + [
+        fname[:-4] + '-%d.fif' % ii for ii in range(1, n + 2)]
+    bad_fname = next_fnames.pop(-1)
+    for ii, this_fname in enumerate(next_fnames[:-1]):
+        assert op.isfile(this_fname), f'Missing file: {this_fname}'
+        with open(this_fname, 'r') as fid:
+            fid.seek(0, 2)
+            file_size = fid.tell()
+        min_ = 0.1 if ii < len(next_fnames) - 1 else 0.1
+        assert size * min_ < file_size <= size, f'{this_fname}'
+    assert not op.isfile(bad_fname), f'Errantly wrote {bad_fname}'
+
+
 def test_epochs_proj(tmpdir):
     """Test handling projection (apply proj in Raw or in Epochs)."""
     tempdir = str(tmpdir)