Repository: biopython/biopython

Issue:
Bio.Entrez.Parser.DataHandler (and presumably other modules) have no way to change local storage directories
This issue came up as a result of [this](http://stackoverflow.com/questions/39147935/errno-13-permission-denied-with-django-on-a-directory-i-dont-want-to-use) and [this](http://stackoverflow.com/questions/30730299/biopython-entrez-wanting-to-access-root-config-in-django-app) question on Stack Overflow.

Briefly, when calling `Bio.Entrez.parse()`, it imports the `DataHandler` class from `.Parser`. The [first several lines](https://github.com/biopython/biopython/blob/9efd8c0e9f8fcd73c3c9b62a280912c6f5a2c9cd/Bio/Entrez/Parser.py#L166-L176) generate a `biopython` directory and `Bio.Entrez.DTDs` and `Bio.Entrez.XSDs` inside it - on Windows it's `%APPDATA%\biopython`, and on Linux/Unix/OS X it's in `~/.config`. This is all fine and dandy if you're running Biopython as a regular user, but the two linked questions above involve cases where people are running the module under Django (the same issue could also occur with other frameworks) which runs as some sort of system user, like `apache` or `httpd` or something like that (one was using `root` for some reason, for goodness sake), which don't necessarily have home directories assigned, or don't have write permission to them, which means the code will error out and not run.

Unfortunately, there is no current way (that I am aware of) reflected in the code to override these default settings, possibly to `/tmp` or some other specified directory. This override could take the form of an environment variable, a config file, or even a parameter passed in the code, similar to how `Bio.Entrez.email` works. All three would be the most flexible, although you'd have to figure out the hierarchy and what to do if two or more conflict. If I had to pick one, a function that checks for an environment variable and uses it unless a code parameter is defined would be best.

Additionally, I would suggest that all of the directory setup code be transferred to the `__init__()` method, as that seems like the best place for it to be, instead of just sitting at the top of the class, outside of any method or property or whatnot.



Discussion:
Exposing the path as a module level attribute (like `Bio.Entrez.email`) sounds sensible, what do you think @mdehoon?

@peterjc , @MattDMo  You can access these paths as follows:

``` python
>>> from Bio.Entrez.Parser import DataHandler
>>> DataHandler.global_dtd_dir
'/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/Bio/Entrez/DTDs'
>>> DataHandler.local_dtd_dir
'/Users/mdehoon/.config/biopython/Bio/Entrez/DTDs'
```

Thanks @mdehoon - That looks to be enough for this specific need...

Not if this code was in the class `__init__` then you could only change it for a single instance of the class.

@MattDMo does this work for you?

For anyone else who might be blocked by this issue, here is my workaround until this issue is resolved in biopython. It copies `Entrez.read` as-is and sets the parsers directory to `/tmp`; change as per your needs:
```python
# patched version of Entrez.read
def _Entrez_read(handle, validate=True, escape=False):
    from Bio.Entrez import Parser
    handler = Entrez.Parser.DataHandler(validate, escape)
    handler.directory = '/tmp' # the only difference between this and `Entrez.read`
    record = handler.read(handle)
    return record

result = _Entrez_read(Entrez.esearch(db='gene', term='IDH1'))
```
@amirkdv try this instead if you want to change the user's cache:

```python
from Bio.Entrez.Parser import DataHandler
DataHandler.local_dtd_dir = "/tmp"
```

[*Update: That idea was later reported not work*]

However, the far simpler workaround is to empty the cache and thus get rid of any out of date DTD or XSD file. i.e.

```bash
$ rm ~/.config/biopython/Bio/Entrez/DTDs/*.dtd
$ rm ~/.config/biopython/Bio/Entrez/XSDs/*.xsd
```

@mdehoon cross reference https://github.com/biopython/biopython/issues/1848#issuecomment-497361760 - it looks like the default value is not initialised just by directly creating the object and trying to access the property:

```
$ python -c "from Bio.Entrez.Parser import DataHandler; print(DataHandler.local_dtd_dir)"

```

i.e. it returns None. Should the directory property get-function or ``__init__`` being calling ``_initialize_directory()``, which would support this use-case?
@peterjc Also I think the relevant method `DataHandler._initialize_directory` seems to not honor the class attribute `local_dtd_dir`; it overwrites it with a path based on `self.directory`, see [code](https://github.com/biopython/biopython/blob/44ecbad7b2e847eb5de6d08e921078f61a15011a/Bio/Entrez/Parser.py#L980).
@peterjc To be clear, I have tried the solution you suggested (biopython version 1.73) and still get the same issue: the working directory ends up being whatever is referenced in my link above.
Does my recommended workaround of emptying the default local cache work though? i.e.

```bash
$ rm ~/.config/biopython/Bio/Entrez/DTDs/*.dtd
$ rm ~/.config/biopython/Bio/Entrez/XSDs/*.xsd
```

P.S. Which version of Python are you using?
@meridiajane confirms on #4732 that this idea does not work:

```python
from Bio.Entrez.Parser import DataHandler
DataHandler.local_dtd_dir = "/tmp"
```
I've also just tried this:
```

# patched version of Entrez.read
def _Entrez_read(handle, validate=True, escape=False):
    from Bio.Entrez import Parser
    handler = Entrez.Parser.DataHandler(validate, escape)
    handler.directory = '/tmp' # the only difference between this and `Entrez.read`
    record = handler.read(handle)
    return record

result = _Entrez_read(Entrez.esearch(db='gene', term='IDH1'))
```

And I don't know if I've been doing something wrong, but that one still failed because importing Parser attempts to make the directory, so it doesn't make it to the `handler.directory = '/tmp'` stage before failing because it lacks permissions to make the directory.
Removing the import for Parser and trying to define the directory without it (because I assume it should still have Parser as part of importing Entrez?) fails because Parser is not an attribute of Bio.Entrez
So in summary, none of the workaround ideas work on a non-Windows setup without a writable home directory. i.e. This probably needs a code change @mdehoon?

(On Windows the code looks at the ``$APPDATA`` setting instead)
It doesn't feel like it, though if these were all meant to work on a windows setup that leaves me feeling a bit more sane. I was convinced I had done something wrong if no one else had had these issues haha.

(For context, I'm trying to run in a docker container running debian, biopython 1.83, and python 3.8.18. If it's a versioning issue I could try and update)

Though I am confused about why it seems to fail when importing parser for me while other posts mention changing it after the fact? Is making the directory at the time of importing parser not the way it has been/should be running?
@meridiajane  The earlier posts are from 2016, so the Biopython code may have changed since then.

> So in summary, none of the workaround ideas work on a non-Windows setup without a writable home directory. i.e. > This probably needs a code change @mdehoon?

I think so, yes.
@meridiajane  Would you be able to suggest a fix and create a PR?
I think so! I'm not quite sure where [Parser.py](https://github.com/biopython/biopython/blob/master/Bio/Entrez/Parser.py) gets `value` at line 331, but assuming its `cls._directory` (line 323), which is used when there's a permission error for the default directory, then couldn't we change:
```
        except PermissionError:
            cls._directory = None  # no local cache
```
To:
```
        except PermissionError:
            cls._directory = Entrez.cache_dir  # no local cache
```
and make Entrez.cache_dir something we can define after importing Entrez (Somewhere in the block of variables at [Entrez's](https://github.com/biopython/biopython/blob/master/Bio/Entrez/__init__.py) lines 140-144)? It'd mean we'd have to `import Entrez.cache_dir` and delete it after making the directories, but I think it'd fix the problem?

If so, I can go ahead and pull+implement that fix
Defining ``Bio.Entrez.local_cache`` or similar (defaulting to None, document that the user can set this to a path) makes sense to me - very similar to the API to set the email or tool name for the Entrez calls. This new setting could be used in preference to the current path (if not None)?

Resolving patch:
diff --git a/Bio/Entrez/Parser.py b/Bio/Entrez/Parser.py
--- a/Bio/Entrez/Parser.py
+++ b/Bio/Entrez/Parser.py
@@ -317,10 +317,13 @@ class DataHandlerMeta(type):
 
     def __init__(cls, *args, **kwargs):
         """Initialize the class."""
+        from Bio import Entrez
+
         try:
-            cls.directory = None  # use default directory for local cache
+            cls.directory = Entrez.local_cache  # use default directory for local cache
         except PermissionError:
-            cls._directory = None  # no local cache
+            cls._directory = Entrez.local_cache  # no local cache
+        del Entrez
 
     @property
     def directory(cls):
diff --git a/Bio/Entrez/__init__.py b/Bio/Entrez/__init__.py
--- a/Bio/Entrez/__init__.py
+++ b/Bio/Entrez/__init__.py
@@ -142,6 +142,7 @@
 sleep_between_tries = 15
 tool = "biopython"
 api_key = None
+local_cache = None
 
 
 # XXX retmode?


Test patch: