BUG: lzma is a required part of python, make it optional
State: closed

Importing `lzma` when Python has been compiled without its support (usually due to a lack of `xz` in the system) will raise a warning. Substituted `import lzma` for helper function.

- [x] closes #27575
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Labels:
name: IO Data
description: IO issues that don't fit into a more specific label

Comments:
I cannot modify the `pandas/_libs/parsers.pyx` without running into issues with `cythonize` (via `pip install -e .`).
> I cannot modify the pandas/_libs/parsers.pyx without running into issues with cythonize (via pip install -e .).

What issues do you see? You'll need a C compiler, since you're updating a Cython file. Typically I use `python setup.py build_ext --inplace`, but `pip install -e .` may do the same.
> > I cannot modify the pandas/_libs/parsers.pyx without running into issues with cythonize (via pip install -e .).
> 
> What issues do you see? You'll need a C compiler, since you're updating a Cython file. Typically I use `python setup.py build_ext --inplace`, but `pip install -e .` may do the same.

I was adding an `if lzma is None` to one of the lines to see if `RuntimeError` is necessary.
However, when I run `pip install -e .` after modifying the file, I get:
```
    ERROR: Complete output from command python setup.py egg_info:
    ERROR: 
    Error compiling Cython file:
    ------------------------------------------------------------
    ...
    
    
    cdef extern from "parser/tokenizer.h":
    
        ctypedef enum ParserState:
    	START_RECORD
    ^
    ------------------------------------------------------------
    
    pandas/_libs/parsers.pyx:90:0: Mixed use of tabs and spaces
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/Users/guilhermesalome/Desktop/pandas/setup.py", line 813, in <module>
        ext_modules=maybe_cythonize(extensions, compiler_directives=directives),
      File "/Users/guilhermesalome/Desktop/pandas/setup.py", line 542, in maybe_cythonize
        return cythonize(extensions, *args, **kwargs)
      File "/Users/guilhermesalome/.pyenv/versions/miniconda3-latest/envs/pandas-dev/lib/python3.7/site-packages/Cython/Build/Dependencies.py", line 1096, in cythonize
        cythonize_one(*args)
      File "/Users/guilhermesalome/.pyenv/versions/miniconda3-latest/envs/pandas-dev/lib/python3.7/site-packages/Cython/Build/Dependencies.py", line 1219, in cythonize_one
        raise CompileError(None, pyx_file)
    Cython.Compiler.Errors.CompileError: pandas/_libs/parsers.pyx
    Compiling pandas/_libs/parsers.pyx because it changed.
    [1/1] Cythonizing pandas/_libs/parsers.pyx
    ----------------------------------------
ERROR: Command "python setup.py egg_info" failed with error code 1 in /Users/guilhermesalome/Desktop/pandas/
```
I am modifying the file in Emacs without any mode for Cython.
The error shows up at a line that has nothing to do with the modification I made.
I managed to make it work in `nano` and putting whitespaces by hand. It might be an issue with my Emacs using tabs.
```
+def test_with_missing_lzma():
+    # https://github.com/pandas-dev/pandas/issues/27575
+    code = textwrap.dedent(
+        """\
+    import sys
+    sys.modules['lzma'] = None
+    import pandas
+    """
+    )
+    subprocess.check_output(["python", "-c", code])
```

This is much smarter than what I was doing to test the code (change the name of the `lzma.py` module).
I do not understand what the command `git diff upstream/master -u -- "*.py" | flake8 --diff` does, but if I do `git diff master -u -- "*.py" | flake8 --diff"` the output is nothing.
Please let me know if this is looking good, and how I can improve the solution.
>  but if I do git diff master -u -- "*.py" | flake8 --diff" the output is nothing

That means it is passing that check.

The isort check is failing.  Should be fixed if you run `isort -rc pandas`
@Salompas it's an isort issue: https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=15881&view=logs&jobId=435ca956-9126-505f-566f-fff31072e2ba&taskId=e3717827-2a2f-5148-8ad4-7900d2171f6e&lineStart=37&lineEnd=38&colStart=1&colEnd=1

https://dev.pandas.io/development/contributing.html#import-formatting
> @Salompas it's an isort issue: https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=15881&view=logs&jobId=435ca956-9126-505f-566f-fff31072e2ba&taskId=e3717827-2a2f-5148-8ad4-7900d2171f6e&lineStart=37&lineEnd=38&colStart=1&colEnd=1
> 
> https://dev.pandas.io/development/contributing.html#import-formatting

Thanks for the link, I will fix the issue.
I made the changes you suggested, let me know how it is looking now.
> Looks pretty close. Can you
> 
> 1. Rename `import_lzma` to `_import_lzma` (`pandas.compat` is supposed to be private, but people still use it)
> 2. move the `if lzma is None` check to a new method in `pandas/compat/__init__.py` called `_get_lzma_file(lzma)`. If `lzma` is None it raises the RuntimeError. Otherwise, it returns `lzma.LZMAFile`, which we can pass the kwargs to when lzma is not None.
> 3. Add at least one subprocess-style test that hits the RuntimeError.

Done.
1., 2. - Good idea! Looks way better now.
3. - The test will fail if you do not have `lzma`, but succeeds otherwise.
Just for my future reference.
On `Emacs` if you have the following in your `init.el`:
```
(progn (add-hook 'before-save-hook 'whitespace-cleanup)
       (add-hook 'before-save-hook (lambda() (delete-trailing-whitespace))))
```
When saving a file, the function `whitespace-cleanup` is called and it  substitutes spaces for tabs whenever possible.
To change the behavior so that tabs are substituted for space instead, add the following to `init.el`:
`(setq-default indent-tabs-mode nil)
`
The variable `indent-tabs-mode` is used by `whitespace-cleanup` and will change its behavior.
Now there should not be any issues with `pyx` files.
I guess another modification to this code is to show the warning just once (right now it shows up twice), but this might not be worth the time.
@jreback @TomAugspurger 
Thanks for the suggestions. I made the changes to the code.
Could you check and let me know where to improve? Thanks!
Thanks @Salompas!
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 0.25.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 ba94f9baaa1a802eee4820a1188db6501a231ae6
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #27882: BUG: lzma is a required part of python, make it optional'
```

4. Push to a named branch :

```
git push YOURFORK 0.25.x:auto-backport-of-pr-27882-on-0.25.x
```

5. Create a PR against branch 0.25.x, I would have named this PR:

> "Backport PR #27882 on branch 0.25.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
Backported in #28012.
I opened https://github.com/pandas-dev/pandas/issues/43461 to re-visit this. I would like to propose to remove the import warning if lzma is missing. Please comment in the linked issue if you disagree.