Repository: mne-tools/mne-python

Issue:
ENH: Add preload=str to concatenate_raw and concatenate_epochs
Hello all, I am using mne.concatenate_raws to collect all my raw data for a single subject. This often caps out my memory on my computer. Is there a way to better manage memory in this situation. Another potential solution is to concatenate epochs which is much better for memory. However, I am concerned that filtering short epochs would cause other issues in my data.

@larsoner has suggested the following:
can you open an issue about concatenate_raws supporting preload as a string?
basically it would allow using a memmap'ed file
in the meantime you can filter each raw instance and save it to disk
and load each of those filtered ones with preload=False (don't load into memory)
then do concatenate_raws on those
then do mne.Epochs on the concatenated instance
the epochs will pull data from the concatenated instance, which will pull them from disk instead of memory




Discussion:

Resolving patch:
diff --git a/mne/io/base.py b/mne/io/base.py
--- a/mne/io/base.py
+++ b/mne/io/base.py
@@ -1516,6 +1516,7 @@ def load_bad_channels(self, bad_file=None, force=False):
         else:
             self.info['bads'] = []
 
+    @fill_doc
     def append(self, raws, preload=None):
         """Concatenate raw instances as if they were continuous.
 
@@ -1529,14 +1530,7 @@ def append(self, raws, preload=None):
         raws : list, or Raw instance
             List of Raw instances to concatenate to the current instance
             (in order), or a single raw instance to concatenate.
-        preload : bool, str, or None (default None)
-            Preload data into memory for data manipulation and faster indexing.
-            If True, the data will be preloaded into memory (fast, requires
-            large amount of memory). If preload is a string, preload is the
-            file name of a memory-mapped file which is used to store the data
-            on the hard drive (slower, requires less memory). If preload is
-            None, preload=True or False is inferred using the preload status
-            of the raw files passed in.
+        %(preload_concatenate)s
         """
         if not isinstance(raws, list):
             raws = [raws]
@@ -2125,10 +2119,7 @@ def concatenate_raws(raws, preload=None, events_list=None, verbose=None):
     ----------
     raws : list
         List of Raw instances to concatenate (in order).
-    preload : bool, or None
-        If None, preload status is inferred using the preload status of the
-        raw files passed in. True or False sets the resulting raw file to
-        have or not have data preloaded.
+    %(preload_concatenate)s
     events_list : None | list
         The events to concatenate. Defaults to None.
     %(verbose)s
diff --git a/mne/utils/docs.py b/mne/utils/docs.py
--- a/mne/utils/docs.py
+++ b/mne/utils/docs.py
@@ -37,6 +37,16 @@
     large amount of memory). If preload is a string, preload is the
     file name of a memory-mapped file which is used to store the data
     on the hard drive (slower, requires less memory)."""
+docdict['preload_concatenate'] = """
+preload : bool, str, or None (default None)
+    Preload data into memory for data manipulation and faster indexing.
+    If True, the data will be preloaded into memory (fast, requires
+    large amount of memory). If preload is a string, preload is the
+    file name of a memory-mapped file which is used to store the data
+    on the hard drive (slower, requires less memory). If preload is
+    None, preload=True or False is inferred using the preload status
+    of the instances passed in.
+"""
 
 # Cropping
 docdict['include_tmax'] = """


Test patch: