Repository: NeurodataWithoutBorders/pynwb

Issue:
[Bug]: Fail to import in read-only file system
### What happened?

I'm trying to import `pynwb` within an Apptainer container and I'm getting a Read-only filesystem error:
```
apptainer shell /allen/aind/scratch/alessio.buccino/singularity_cache_dir/ghcr.io-allenneuraldynamics-aind-ephys-pipeline-nwb-si-0.102.1.img
```
In the container
```
Apptainer> python
Python 3.11.5 (main, Sep 11 2023, 13:54:46) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pynwb
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/conda/lib/python3.11/site-packages/pynwb/__init__.py", line 250, in <module>
    __load_core_namespace()
  File "/opt/conda/lib/python3.11/site-packages/pynwb/__init__.py", line 219, in __load_core_namespace
    Path(__resources['cached_typemap_path']).unlink(missing_ok=True)
  File "/opt/conda/lib/python3.11/pathlib.py", line 1147, in unlink
    os.unlink(self)
OSError: [Errno 30] Read-only file system: '/opt/conda/lib/python3.11/site-packages/pynwb/core_typemap.pkl'
```

I think that the error might be related to running in an environment where I previously used an apptainer image with another pynwb version.

Anyways, I think that the code should have some try-except to avoid errors when unlinking the core_typemap

### Steps to Reproduce

```python
import pynwb # in an apptainer container
```

### Traceback

```python
Apptainer> python
Python 3.11.5 (main, Sep 11 2023, 13:54:46) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pynwb
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/conda/lib/python3.11/site-packages/pynwb/__init__.py", line 250, in <module>
    __load_core_namespace()
  File "/opt/conda/lib/python3.11/site-packages/pynwb/__init__.py", line 219, in __load_core_namespace
    Path(__resources['cached_typemap_path']).unlink(missing_ok=True)
  File "/opt/conda/lib/python3.11/pathlib.py", line 1147, in unlink
    os.unlink(self)
OSError: [Errno 30] Read-only file system: '/opt/conda/lib/python3.11/site-packages/pynwb/core_typemap.pkl'
```

### Operating System

Linux

### Python Executable

Conda

### Python Version

3.11

### Package Versions

pynwb == 2.8.3

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/NeurodataWithoutBorders/pynwb/blob/dev/.github/CODE_OF_CONDUCT.rst)
- [x] Have you checked the [Contributing](https://github.com/NeurodataWithoutBorders/pynwb/blob/dev/docs/CONTRIBUTING.rst) document?
- [x] Have you ensured this bug was not already [reported](https://github.com/NeurodataWithoutBorders/pynwb/issues)?

Discussion:
codecov[bot]: ## [Codecov](https://app.codecov.io/gh/NeurodataWithoutBorders/pynwb/pull/2057?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NeurodataWithoutBorders) Report
Attention: Patch coverage is `92.00000%` with `2 lines` in your changes missing coverage. Please review.
> Project coverage is 83.00%. Comparing base [(`01d9ab9`)](https://app.codecov.io/gh/NeurodataWithoutBorders/pynwb/commit/01d9ab92de25dcd465cfad6d68d9df54679ea54f?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NeurodataWithoutBorders) to head [(`f9a2322`)](https://app.codecov.io/gh/NeurodataWithoutBorders/pynwb/commit/f9a23221a2c37a2fc925c3126f963fcc83551ec3?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NeurodataWithoutBorders).

| [Files with missing lines](https://app.codecov.io/gh/NeurodataWithoutBorders/pynwb/pull/2057?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NeurodataWithoutBorders) | Patch % | Lines |
|---|---|---|
| [src/pynwb/\_\_init\_\_.py](https://app.codecov.io/gh/NeurodataWithoutBorders/pynwb/pull/2057?src=pr&el=tree&filepath=src%2Fpynwb%2F__init__.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NeurodataWithoutBorders#diff-c3JjL3B5bndiL19faW5pdF9fLnB5) | 92.00% | [2 Missing :warning: ](https://app.codecov.io/gh/NeurodataWithoutBorders/pynwb/pull/2057?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NeurodataWithoutBorders) |

> :exclamation:  There is a different number of reports uploaded between BASE (01d9ab9) and HEAD (f9a2322). Click for more details.
> 
> <details><summary>HEAD has 15 uploads less than BASE</summary>
>
>| Flag | BASE (01d9ab9) | HEAD (f9a2322) |
>|------|------|------|
>|unit|8|1|
>|integration|8|0|
></details>

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##              dev    #2057      +/-   ##
==========================================
- Coverage   92.84%   83.00%   -9.85%     
==========================================
  Files          28       28              
  Lines        2755     2766      +11     
  Branches      716      714       -2     
==========================================
- Hits         2558     2296     -262     
- Misses        127      372     +245     
- Partials       70       98      +28     
```

| [Flag](https://app.codecov.io/gh/NeurodataWithoutBorders/pynwb/pull/2057/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NeurodataWithoutBorders) | Coverage Δ | |
|---|---|---|
| [integration](https://app.codecov.io/gh/NeurodataWithoutBorders/pynwb/pull/2057/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NeurodataWithoutBorders) | `?` | |
| [unit](https://app.codecov.io/gh/NeurodataWithoutBorders/pynwb/pull/2057/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NeurodataWithoutBorders) | `83.00% <92.00%> (+0.24%)` | :arrow_up: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NeurodataWithoutBorders#carryforward-flags-in-the-pull-request-comment) to find out more.

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/NeurodataWithoutBorders/pynwb/pull/2057?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NeurodataWithoutBorders).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NeurodataWithoutBorders).

<details><summary>🚀 New features to boost your workflow: </summary>

- ❄ [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
rly: This all looks good to me! Great fix.

Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index c1430ea4..d884c0d5 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -2,6 +2,9 @@
 
 ## Upcoming
 
+### Enhancements and minor changes
+- Added option to disable typemap caching and updated type map cache location. @stephprince [#2057](https://github.com/NeurodataWithoutBorders/pynwb/pull/2057)
+
 ### Bug fixes
 - Fix `add_data_interface` functionality that was mistakenly removed in PyNWB 3.0. @stephprince [#2052](https://github.com/NeurodataWithoutBorders/pynwb/pull/2052)
 
diff --git a/docs/gallery/general/plot_read_basics.py b/docs/gallery/general/plot_read_basics.py
index df138e29..556b7c10 100644
--- a/docs/gallery/general/plot_read_basics.py
+++ b/docs/gallery/general/plot_read_basics.py
@@ -82,7 +82,7 @@ download("https://api.dandiarchive.org/api/assets/0f57f0b0-f021-42bb-8eaa-56cd48
 # .. seealso::
 #
 #   Learn about all the different ways you can download data from the DANDI Archive
-#   `here <https://docs.dandiarchive.org/12_download/>`_
+#   `here <https://docs.dandiarchive.org/user-guide-using/accessing-data/downloading/>`_
 #
 # .. seealso:: Streaming data
 #
diff --git a/docs/source/install_developers.rst b/docs/source/install_developers.rst
index ba964ff2..cfc2c34e 100644
--- a/docs/source/install_developers.rst
+++ b/docs/source/install_developers.rst
@@ -86,6 +86,11 @@ editable mode.
     pip install -r requirements.txt -r requirements-dev.txt
     pip install -e .
 
+.. note::
+    When importing PyNWB for the first time, the type map is automatically cached in a pickle file to improve load 
+    times on future imports. However, this can cause issues if you are modifying the schema and testing schema-related 
+    changes. To disable type map caching, set the environment variable ``PYNWB_NO_CACHE_DIR`` to ``1``.
+
 
 Run tests
 ---------
diff --git a/pyproject.toml b/pyproject.toml
index 2d3f54f6..10e4c739 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -38,6 +38,7 @@ dependencies = [
     "numpy>=1.24.0",
     "pandas>=1.2.0",
     "python-dateutil>=2.8.2",
+    "platformdirs>=4.1.0"
 ]
 dynamic = ["version"] # the build backend will compute the version dynamically from git tag (or a __version__)
 
diff --git a/requirements-min.txt b/requirements-min.txt
index c9dc53ee..a9f3bcb7 100644
--- a/requirements-min.txt
+++ b/requirements-min.txt
@@ -4,3 +4,4 @@ hdmf==3.14.5
 numpy==1.24.0
 pandas==1.2.0
 python-dateutil==2.8.2
+platformdirs==4.1.0
\ No newline at end of file
diff --git a/requirements.txt b/requirements.txt
index ad3152f3..16936de2 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -5,3 +5,4 @@ numpy==2.1.1; python_version > "3.9"  # numpy 2.1+ is not compatible with py3.9
 numpy==2.0.2; python_version == "3.9"
 pandas==2.2.3
 python-dateutil==2.9.0.post0
+platformdirs==4.3.6
diff --git a/src/pynwb/__init__.py b/src/pynwb/__init__.py
index 60656e41..5adef554 100644
--- a/src/pynwb/__init__.py
+++ b/src/pynwb/__init__.py
@@ -7,6 +7,7 @@ from copy import deepcopy
 import subprocess
 import pickle
 from warnings import warn
+from platformdirs import PlatformDirs
 import h5py
 
 from hdmf.spec import NamespaceCatalog
@@ -69,13 +70,16 @@ def __get_resources() -> dict:
     __location_of_this_file = files(__name__)
     __core_ns_file_name = 'nwb.namespace.yaml'
     __schema_dir = 'nwb-schema/core'
-    cached_core_typemap = __location_of_this_file / 'core_typemap.pkl'
-    cached_version_indicator = __location_of_this_file / '.core_typemap_version'
+
+    # create a cache directory for the core typemap
+    dirs = PlatformDirs(appname="pynwb", version=__version__, ensure_exists=True)
+    cache_dir = dirs.user_cache_path
+    cached_core_typemap = cache_dir / 'pynwb_core_typemap.pkl'
 
     ret = dict()
     ret['namespace_path'] = str(__location_of_this_file / __schema_dir / __core_ns_file_name)
     ret['cached_typemap_path'] = str(cached_core_typemap)
-    ret['cached_version_indicator'] = str(cached_version_indicator)
+    ret['user_cache_dir'] = str(dirs.user_cache_dir)
     return ret
 
 
@@ -149,6 +153,7 @@ def load_namespaces(**kwargs):
     namespace_path = getargs('namespace_path', kwargs)
     return __TYPE_MAP.load_namespaces(namespace_path)
 
+
 def available_namespaces():
     """Returns all namespaces registered in the namespace catalog"""
     return __TYPE_MAP.namespace_catalog.namespaces
@@ -180,6 +185,15 @@ def __clone_submodules():
     else:  # pragma: no cover
         raise RuntimeError("Package is not installed from a git repository, can't clone submodules")
 
+def clear_cache_dir():
+    """
+    Clear all the cached typemap files for all versions
+    """
+    try:
+        for f in Path(__resources['user_cache_dir']).parent.glob('**/pynwb_core_typemap.pkl'):
+            Path(f).unlink()
+    except (OSError, PermissionError) as e:
+        warn(f'Could not clear cache directory: {e}', UserWarning)
 
 def __load_core_namespace(final:bool=False):
     """
@@ -197,30 +211,25 @@ def __load_core_namespace(final:bool=False):
     global __TYPE_MAP
     global __resources
 
-    # if we have a version indicator file and it doesn't match the current version,
-    # scrap the cached typemap
-    if os.path.exists(__resources['cached_version_indicator']):
-        with open(__resources['cached_version_indicator'], 'r') as f:
-            cached_version = f.read().strip()
-        if cached_version != __version__:
-            Path(__resources['cached_typemap_path']).unlink(missing_ok=True)
-    else:
-        # remove any cached typemap, forcing re-creation
-        Path(__resources['cached_typemap_path']).unlink(missing_ok=True)
+    # Check for environment variable to disable caching
+    disable_cache = os.environ.get('PYNWB_NO_CACHE_DIR', '0') == '1'
 
-    # load pickled typemap if we have one
-    if os.path.exists(__resources['cached_typemap_path']):
+    # load pickled typemap if we have one and want to use caching
+    if os.path.exists(__resources['cached_typemap_path']) and not disable_cache:
         with open(__resources['cached_typemap_path'], 'rb') as f:
             __TYPE_MAP = pickle.load(f)  # type: TypeMap
 
     # otherwise make a new one and cache it
     elif os.path.exists(__resources['namespace_path']):
         load_namespaces(__resources['namespace_path'])
-        with open(__resources['cached_typemap_path'], 'wb') as f:
-            pickle.dump(__TYPE_MAP, f, protocol=pickle.HIGHEST_PROTOCOL)
-        with open(__resources['cached_version_indicator'], 'w') as f:
-            f.write(__version__)
 
+        # cache the loaded namespace if caching is not disabled
+        if not disable_cache:
+            try:
+                with open(__resources['cached_typemap_path'], 'wb') as f:
+                    pickle.dump(__TYPE_MAP, f, protocol=pickle.HIGHEST_PROTOCOL)
+            except (OSError, PermissionError):
+                pass  # skip caching if we can't write the cached typemap
     # otherwise, we don't have the schema and try and initialize from submodules,
     # afterwards trying to load the namespace again
     else:
@@ -634,6 +643,7 @@ __all__ = [
     'get_manager',
     'load_namespaces', 
     'available_namespaces',
+    'clear_cache_dir',
     'register_class',
     'register_map',
     'get_class',


Test patch:
diff --git a/tests/unit/test_typemap_cache.py b/tests/unit/test_typemap_cache.py
new file mode 100644
index 00000000..b0b93428
--- /dev/null
+++ b/tests/unit/test_typemap_cache.py
@@ -0,0 +1,60 @@
+import os
+import tempfile
+import pynwb
+import unittest
+from pathlib import Path
+from unittest.mock import patch
+
+
+class TestTypemapCache(unittest.TestCase):
+    """Tests for the typemap caching functionality."""
+
+    def setUp(self):
+        """Set up a temporary directory for cache files."""
+
+        # create mock resources variables for testing
+        self.temp_dir = tempfile.TemporaryDirectory()
+        self.original_resources = getattr(pynwb, '__resources')
+        self.mock_resources = dict(
+            namespace_path=self.original_resources['namespace_path'],
+            user_cache_dir=Path(self.temp_dir.name) / "pynwb" / pynwb.__version__,
+            cached_typemap_path=Path(self.temp_dir.name) / "pynwb" / pynwb.__version__ / 'pynwb_core_typemap.pkl',
+        )
+
+        # make the cache directories if they do not exist
+        Path(self.mock_resources['user_cache_dir']).mkdir(parents=True, exist_ok=True)
+
+    def tearDown(self):
+        """Clean up temporary directory and restore environment."""
+        self.temp_dir.cleanup()
+
+    @patch.dict(os.environ, {'PYNWB_NO_CACHE_DIR': '1'})
+    def test_typemap_cache_disabled(self):
+        """Test that caching is disabled when environment variable is set."""
+        with patch.object(pynwb, '__resources', self.mock_resources):
+            # Remove any existing cache files before we start, make the cache_dir
+            cache_path = Path(self.mock_resources['cached_typemap_path'])
+            cache_path.unlink(missing_ok=True)
+
+            # Force a reload of the core namespace
+            load_func = getattr(pynwb, '__load_core_namespace')
+            load_func()
+
+            # Verify the cache file was not created
+            self.assertFalse(cache_path.exists())
+
+    @patch.dict(os.environ, {'PYNWB_NO_CACHE_DIR': '0'})
+    def test_typemap_cache_error_handling(self):
+        """Test error handling when encountering write errors to cache directory"""
+        with patch.object(pynwb, '__resources', self.mock_resources):
+            # Remove any existing cache file
+            cache_path = Path(self.mock_resources['cached_typemap_path'])
+            cache_path.unlink(missing_ok=True)
+
+            # Mock open to raise a permission error when trying to write
+            with patch('builtins.open', side_effect=PermissionError("Permission denied")):
+                load_func = getattr(pynwb, '__load_core_namespace')
+                load_func()
+
+            # check that cache file was not written
+            self.assertFalse(cache_path.exists())