Repository: pandas-dev/pandas
Pull request: BUG: Fix lxml import in show_versions
Fixed lxml import issue

- [x] closes #23934
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

Problem statement:
show_versions fails on master with latest lxml
#### Code Sample, a copy-pastable example if possible

```python
[ins] In [1]: import pandas as pd
p
[ins] In [2]: pd.show_versions()
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-2-3d232a07e144> in <module>
----> 1 pd.show_versions()

~/workspace/pandas/pandas/util/_print_versions.py in show_versions(as_json)
    107             else:
    108                 mod = importlib.import_module(modname)
--> 109             ver = ver_f(mod)
    110             deps_blob.append((modname, ver))
    111         except ImportError:

~/workspace/pandas/pandas/util/_print_versions.py in <lambda>(mod)
     86         ("xlwt", lambda mod: mod.__VERSION__),
     87         ("xlsxwriter", lambda mod: mod.__version__),
---> 88         ("lxml", lambda mod: mod.etree.__version__),
     89         ("bs4", lambda mod: mod.__version__),
     90         ("html5lib", lambda mod: mod.__version__),

AttributeError: module 'lxml' has no attribute 'etree'
```

#### Expected Output

This seems to work OK on 0.23.4 (output below).

Let me know if there are other potential factors you'd like me to check

#### Output of ``pd.show_versions()``

<details>

[ins] In [5]: pd.show_versions()

INSTALLED VERSIONS
------------------
commit: None
python: 3.7.1.final.0
python-bits: 64
OS: Darwin
OS-release: 18.0.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.23.4
pytest: 3.9.2
pip: 18.1
setuptools: 40.6.2
Cython: 0.28.5
numpy: 1.15.2
scipy: None
pyarrow: None
xarray: 0.10.9
IPython: 7.1.1
sphinx: None
patsy: 0.5.0
dateutil: 2.7.5
pytz: 2018.5
blosc: None
bottleneck: 1.2.1
tables: None
numexpr: 2.6.8
feather: None
matplotlib: 3.0.0
openpyxl: None
xlrd: 1.1.0
xlwt: 1.3.0
xlsxwriter: None
lxml: 4.2.5
bs4: 4.6.0
html5lib: 1.0.1
sqlalchemy: 1.2.12
pymysql: None
psycopg2: None
jinja2: 2.8.1
s3fs: None
fastparquet: None
pandas_gbq: 0.6.1+2.gd98c621
pandas_datareader: 0.7.0
</details>

Hints from issue discussion:
Are you sure lxml is installed properly? Whats the output of

```python
import lxml.etree
lxml.etree.__version__
```
Apologies - it works now. No idea how what could have caused this.

```

[ins] In [1]: import lxml.etree

[ins] In [2]: lxml.etree.__version__
Out[2]: '4.2.5'

[ins] In [3]: import pandas as pd

[ins] In [4]: pd.show_versions()

INSTALLED VERSIONS
------------------
commit: b7294dd3ec47dffa50f3c8bdf89aa8a01f4494f2
python: 3.7.1.final.0
python-bits: 64
OS: Darwin
OS-release: 18.0.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.19.0+4049.gb7294dd3e
pytest: 3.9.2
pip: 18.1
setuptools: 40.6.2
Cython: 0.28.5
numpy: 1.15.2
scipy: None
pyarrow: None
xarray: 0.10.9
IPython: 7.1.1
sphinx: None
patsy: 0.5.0
dateutil: 2.7.5
pytz: 2018.5
blosc: None
bottleneck: 1.2.1
tables: None
numexpr: 2.6.8
feather: None
matplotlib: 3.0.0
openpyxl: None
xlrd: 1.1.0
xlwt: 1.3.0
xlsxwriter: None
lxml: 4.2.5
bs4: 4.6.0
html5lib: 1.0.1
sqlalchemy: 1.2.12
pymysql: None
psycopg2: None
jinja2: 2.8.1
s3fs: None
fastparquet: None
pandas_gbq: 0.6.1+2.gd98c621
pandas_datareader: None
gcsfs: None
```
Actually - if I don't import `lmxl` initially, then it fails:

```
[ins] In [1]: import pandas as pd

[ins] In [2]: pd.show_versions()
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-2-3d232a07e144> in <module>
----> 1 pd.show_versions()

~/workspace/pandas/pandas/util/_print_versions.py in show_versions(as_json)
    107             else:
    108                 mod = importlib.import_module(modname)
--> 109             ver = ver_f(mod)
    110             deps_blob.append((modname, ver))
    111         except ImportError:

~/workspace/pandas/pandas/util/_print_versions.py in <lambda>(mod)
     86         ("xlwt", lambda mod: mod.__VERSION__),
     87         ("xlsxwriter", lambda mod: mod.__version__),
---> 88         ("lxml", lambda mod: mod.etree.__version__),
     89         ("bs4", lambda mod: mod.__version__),
     90         ("html5lib", lambda mod: mod.__version__),

AttributeError: module 'lxml' has no attribute 'etree'
```
Is this because `etree` isn't imported into `lxml`'s root?

```

[ins] In [1]: import lxml

[ins] In [6]: lxml.etree
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-6-f7059756e878> in <module>
----> 1 lxml.etree

AttributeError: module 'lxml' has no attribute 'etree'

[ins] In [7]: from lxml import etree

[ins] In [8]:
```
I can't replicate this on `master` and latest `lxml`.

I'm also inclined to believe `lxml` was not properly installed.
I installed in an empty docker container to check:

```
# docker run -it python:3 bash                                                                                                                                                                                                                                                          

root@f6dd4625aeb6:/# pip install lxml

Collecting lxml
  Downloading https://files.pythonhosted.org/packages/7a/6b/a3d2d3c3075617edcbfc272d79281e812b1a94dab37923b1d06fdfe2e906/lxml-4.2.5-cp37-cp37m-manylinux1_x86_64.whl (5.8MB)
    100% |████████████████████████████████| 5.8MB 2.6MB/s
Installing collected packages: lxml
Successfully installed lxml-4.2.5

root@f6dd4625aeb6:/# python

Python 3.7.1 (default, Nov 16 2018, 22:26:09)
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.

>>> import lxml
>>> lxml.etree
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'lxml' has no attribute 'etree'
```
That suggests a broken wheel perhaps on `lxml` ?
Are you sure it's not that `etree` isn't imported into `lxml` root? That seems like the simplest explanation? 
https://github.com/lxml/lxml/blob/master/src/lxml/__init__.py
I’d suggest waking through what pandas is doing in show_versions. I suspect that we’re relying on someone else importing lxml.etree and you don’t have that other optional dependency.

Regardless, we should probably just check lxml.etree


________________________________
From: Maximilian Roos <notifications@github.com>
Sent: Monday, November 26, 2018 18:23
To: pandas-dev/pandas
Cc: Tom Augspurger; Comment
Subject: Re: [pandas-dev/pandas] show_versions fails on master with latest lxml (#23934)


Are you sure it's not that etree isn't imported into lxml root? That seems like the simplest explanation?
https://github.com/lxml/lxml/blob/master/src/lxml/__init__.py

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub<https://github.com/pandas-dev/pandas/issues/23934#issuecomment-441850932>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ABQHIihRnxiGCFkN92gyTVIFO1s5AmEHks5uzIYQgaJpZM4Yz70l>.

Reproduced. It seems like openpyxl imports `lxml.etree`, so it's already in `sys.modules`.


```
bash-4.4$ pip install openpyxl
Collecting openpyxl
...
Successfully installed openpyxl-2.5.11
bash-4.4$ python -c "import pandas; pandas.show_versions()"

INSTALLED VERSIONS
------------------
commit: a7767b91539d66741a15b3cb05e31ce6c21613e0
python: 3.7.0.final.0
python-bits: 64
OS: Darwin
OS-release: 17.7.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

```

To fix this, we should just change https://github.com/pandas-dev/pandas/blob/90961f2acad5c2db6aa957e2a79df8668f5b02d1/pandas/util/_print_versions.py#L88 to from `"lxml"` to `"lxml.etree"`.

Commit message: Add test case for show_version

Patch to review:
diff --git a/pandas/tests/util/test_util.py b/pandas/tests/util/test_util.py
index a6cb54ee43909..39a0c3cf7f753 100644
--- a/pandas/tests/util/test_util.py
+++ b/pandas/tests/util/test_util.py
@@ -3,6 +3,7 @@
 from collections import OrderedDict
 import locale
 import os
+import subprocess
 import sys
 from uuid import uuid4
 
@@ -530,3 +531,10 @@ def test_safe_import(monkeypatch):
     monkeypatch.setitem(sys.modules, mod_name, mod)
     assert not td.safe_import(mod_name, min_version="2.0")
     assert td.safe_import(mod_name, min_version="1.0")
+
+
+def test_lxml_import():
+    # See GH 23934
+    # lxml does not have attribute etree exception should not be thrown
+    subprocess.check_call([sys.executable, "-c",
+                           "import pandas; pandas.show_versions()"])
diff --git a/pandas/util/_print_versions.py b/pandas/util/_print_versions.py
index 3016bf04b5258..a5c86c2cc80b3 100644
--- a/pandas/util/_print_versions.py
+++ b/pandas/util/_print_versions.py
@@ -85,7 +85,7 @@ def show_versions(as_json=False):
         ("xlrd", lambda mod: mod.__VERSION__),
         ("xlwt", lambda mod: mod.__VERSION__),
         ("xlsxwriter", lambda mod: mod.__version__),
-        ("lxml", lambda mod: mod.etree.__version__),
+        ("lxml.etree", lambda mod: mod.__version__),
         ("bs4", lambda mod: mod.__version__),
         ("html5lib", lambda mod: mod.__version__),
         ("sqlalchemy", lambda mod: mod.__version__),


Code review comments:
diff_hunk: @@ -530,3 +531,10 @@ def test_safe_import(monkeypatch):
     monkeypatch.setitem(sys.modules, mod_name, mod)
     assert not td.safe_import(mod_name, min_version="2.0")
     assert td.safe_import(mod_name, min_version="1.0")
+
+
+def test_lxml_import():
path: pandas/tests/util/test_util.py
text: @user1:
can you rename to ``test_show_versions()`` and the issue number is not needed here. maybe just a comment that this test validations that show_versions() is accessing version info correctly.

@user2:
FWIW, I don't think this test is necessary. It wouldn't have caught the original bug, unless the environment had lxml but not openpyxl.

We also should be running `show_versions` as part of every build already right? Just for the output in the CI logs?

@user1:
ok, @user2 is right here, let's just remove this test entirely.

@author:
@user1 removed the test case