ENH: Add large file support for read_xml
State: closed

- [X] closes ##45442
- [X] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [X] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [X] Added an entry in the latest `doc/source/whatsnew/v1.5.0.rst` file if fixing a bug or adding a new feature.


Labels:
name: IO XML
description: read_xml, to_xml

Comments:
@mroeschke, I combined tests with tests in original and in dtypes. For this new feature, one URL is tested to raise an error so may never reach the endpoint. I removed any new S3 calls. Given the large size potential, this feature requires only uncompressed file system paths.
@ParfaitG if you can merge master
@ParfaitG can you merge master
lgtm, @mroeschke if any comments.
Awesome, thanks for sticking with it
Awesome feature, already started using it!

What is the purpose of this bit at https://github.com/pandas-dev/pandas/blob/fa7e31b8b19eb03dceebd09d03798363daae07d9/pandas/io/xml.py#L677

This started giving me "TypeError: 'NoneType' object does not support item deletion" on my XML files. The equivalent ElementTree  parser does not have this (only lxml). After removing it, everything is working well, at least on my xml file.

As to attribute parsing, shouldn't we parse only on row_node? Otherwise, another element entirely that's a child of row_node could have the attribute but it may not be what we're looking for.

Maybe this? (untested)

```diff
diff --git a/pandas/io/xml.py b/pandas/io/xml.py
index 181b0fe..8740e8c 100644
--- a/pandas/io/xml.py
+++ b/pandas/io/xml.py
@@ -411,13 +411,14 @@ class _EtreeFrameParser(_XMLFrameParser):
             if event == "start":
                 if curr_elem == row_node:
                     row = {}
+                    for col in self.iterparse[row_node]:
+                        if col in elem.attrib:
+                            row[col] = elem.attrib[col]

             if row is not None:
                 for col in self.iterparse[row_node]:
                     if curr_elem == col:
                         row[col] = elem.text.strip() if elem.text else None
-                    if col in elem.attrib:
-                        row[col] = elem.attrib[col]

             if event == "end":
                 if curr_elem == row_node and row is not None:
@@ -659,22 +660,20 @@ class _LxmlFrameParser(_XMLFrameParser):
             if event == "start":
                 if curr_elem == row_node:
                     row = {}
+                    for col in self.iterparse[row_node]:
+                        if col in elem.attrib:
+                            row[col] = elem.attrib[col]

             if row is not None:
                 for col in self.iterparse[row_node]:
                     if curr_elem == col:
                         row[col] = elem.text.strip() if elem.text else None
-                    if col in elem.attrib:
-                        row[col] = elem.attrib[col]
```
Thank you for your comment, @bailsman. Given this PR is closed, can you raise a potential BUG issue of this new feature with a reproducible example? And possibly ask a QST issue for your attribute parsing question or ask on StackOverflow. Specifically, we need a minimum XML sample and attempted code with desired result.
Before I go off and try to debug this so that I can make a reproducible example, can I just ask you what this line does and what the purpose of this line was intended to be? https://github.com/pandas-dev/pandas/blob/fa7e31b8b19eb03dceebd09d03798363daae07d9/pandas/io/xml.py#L677

It's lxml specific (the elementtree equivalent doesn't have it), and, at least, in my case it seems to work fine without it and throws errors if I keep it. On initial investigation, the problem doesn't occur on small files that I've tested, so it would take some additional debugging effort on my part to figure out how to reproduce it, which I'd rather invest with fuller understanding to save time.
Correct. That is an lxml-specific method where [iterparse docs](https://lxml.de/3.2/parsing.html#iterparse-and-iterwalk) indicate that line intends to *clean up preceding siblings* with stated goal:

> If you have elements with a long list of children in your XML file and want to save more memory during parsing, you can clean up the preceding siblings of the current element