BUG: Fix pandas compatibility with Python installations lacking bzip2 headers
State: closed

- [x] Fixes #53857 
- [x] Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

This simple change makes the `bz2` import optional. Some Python installations do not contain bzip2 headers, so unless you are working with bzip compression, there is no reason for pandas to eagerly load (and require) a `bz2` import.

Labels:
name: IO Data
description: IO issues that don't fit into a more specific label

Comments:
Thanks for the PR.

Can you add a test?
In principle, yes, but because `bz2` is a Python standard library, adding a test for this would require setting up a new GitHub action so that we could install a particular lightweight Python that lacks the `libbzip2` headers. The existing `bz2` tests will already hit the behavior when `bz2` is compiled; what we can't test easily is the inverse.

The current `macOS-latest` image on GitHub actions with a Python 3.7 install lacks the libbzip2 headers. However, apparently this is not intended behavior and will be patched soon: https://github.com/actions/setup-python/issues/682. I noticed this `bz2` requirement only because of that bug in the GitHub action, but I thought I might as well fix it anyways as it seems like libbzip2 isn't always installed.
Maybe we can just check that ``bz2`` is not in ``sys.modules``?
Do you mean rather than catching the `ImportError`? I guess we could but it would be less readable. I didn't even know what `sys.modules` was until you mentioned it and I looked it up.
I meant for the test. Everything else looks good.

I was thinking something like this.
```
>>> import sys
>>> sys.modules.pop('bz2', None) # Remove 'bz2' from sys.modules if it exists
>>> import pandas as pd
>>> sys.modules.pop('bz2', None) # Check for 'bz2' in sys.modules
<module 'bz2' from '/Users/thomasli/opt/miniconda3/lib/python3.9/bz2.py'>
```
(except for the last line, you would use an assert I think).

Does this work?
Ah, I understand. That sounds like it should work. Where should I put that test so as to not interfere with those that require bz2?
> Ah, I understand. That sounds like it should work. Where should I put that test so as to not interfere with those that require bz2?

I think any one of the test files that doesn't already explictly import bz2 with (``import bz2``) should be fine. Maybe ``pandas/tests/io/test_common.py``?
It looks like that one has a compression test:

```python
    @pytest.mark.parametrize("encoding", ["utf-16", "utf-32"])
    @pytest.mark.parametrize("compression_", ["bz2", "xz"])
    def test_warning_missing_utf_bom(self, encoding, compression_):
        """
        bz2 and xz do not write the byte order mark (BOM) for utf-16/32.

        https://stackoverflow.com/questions/55171439

        GH 35681
        """
```
So I guess it would break that test?

I'll do the following for a test FYI:

```python
...
sys.modules.pop("bz2", None)  # Remove 'bz2' from available modules for testing
import pandas as pd
...
from pandas.compat import get_bz2_file

...

def test_bz2_nonimport():
    assert "bz2" not in sys.modules
    msg = "bz2 module not available."
    with pytest.raises(RuntimeError, match=msg):
        get_bz2_file()
```
You can try ``pandas/tests/test_common.py`` as well. 

That should probably be safer.
I think you're missing the first test now, which is to check that bz2 is not in ``sys.modules`` after importing pandas.

Can you add that test (similar to how you're doing the other one now)?
Sorry I'm not sure I understand. Wouldn't bz2 *always* be in sys.modules in this test? We are just setting it to `None`. Why would importing pandas remove bz2 from sys.modules?
Popping bz2 from the modules does not trigger an import error, if that is the point of confusion:

```python
[ins] In [1]: import sys

[ins] In [2]: sys.modules.pop('bz2', None)
Out[2]: <module 'bz2' from '/Users/mcranmer/mambaforge/envs/pandas-dev/lib/python3.10/bz2.py'>

[ins] In [3]: sys.modules['bz2']
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
Cell In[3], line 1
----> 1 sys.modules['bz2']

KeyError: 'bz2'

[ins] In [4]: import pandas as pd

[nav] In [5]: sys.modules['bz2']
Out[5]: <module 'bz2' from '/Users/mcranmer/mambaforge/envs/pandas-dev/lib/python3.10/bz2.py'>
```

Only `sys.modules['bz2'] = None` could trigger the import error.
> I think you're missing the first test now, which is to check that bz2 is not in `sys.modules` after importing pandas.
> 
> Can you add that test (similar to how you're doing the other one now)?

To clarify, I meant to check that bz2 is not in `sys.modules` after importing pandas. If you popped bz2 out of `sys.modules` than it should not appear again after importing pandas.

To avoid messing with the other tests, you should launch a new Python process like the other test.

The way the import error test is done now is fine.
> To clarify, I meant to check that bz2 is not in `sys.modules` after importing pandas. If you popped bz2 out of `sys.modules` than it should not appear again after importing pandas.

This is what I showed in the example above. Even after popping it, it reappears in `sys.module`.
> Even after popping it, it reappears in `sys.module`.

I thought this PR was meant to make it so bz2 is not imported when you do `import pandas as pd` but only when needed. Doesn't this mean importing pandas on a setup that doesn't have bz2 will fail?
> I thought this PR was meant to make it so bz2 is not imported when you do `import pandas as pd` but only when needed. Doesn't this mean importing pandas on a setup that doesn't have bz2 will fail?

No, bz2 is still imported when loading pandas, but now it only happens when bz2 is installed. I did it this way to be consistent with the other optional libraries in pandas (e.g., lzma), but I could change it; let me know.

What happens at runtime now is that it checks whether bz2 was loaded or not before importing the BZ2File class.
Thanks @MilesCranmer