benbernard/RecordStream — pull request #25 by elindsey (2013-12-24 23:34:33+00:00)
Title: switch fromxml from XML::Simple to XML::Twig

Consider this more of a discussion request than a pull request. I don't like the performance regression I introduced and want to sort that out, but I'm interested in your thoughts before I go too far down that path (plus I can only stomach so much XML at a time). :)

Pros:
* Preserves document order, which is both nice to have and makes fromxml.t pass in perl 5.18 (https://rt.cpan.org/Public/Bug/Display.html?id=91452)
* Handles large XML documents (doesn't need to build an entire in-memory representation). A ~100mb XML doc spikes to 2gb for me under XML::Simple, but uses a constant 500mb of RAM under Twig
* Streams records as they are parsed from XML instead of first blocking on converting the entire doc to an in memory structure (though currently the text of the XML doc is still buffered in memory)
* Gives us a way to expose something like --xpath to do more interesting element selection

Fixes a few bugs:
* ForceArray was improperly set to 1 (passing a hashref where an arrayref was expected)
* 'element' entries weren't added when multiple --element were specified, but --nested wasn't set

Cons:
* Up to 50% slower for "dense" XML docs that fit in RAM (docs where nearly every element is converted to a record). Still trying to see if I can bring that down without porting the whole bloomin' thing to SAX and writing directly to a Record...