EOL/eol_php_code — pull request #71 by hyanwong (2013-10-17 10:29:14+00:00)
Title: Wikimedia small bugfixes

Debugging a problem with superfluous newlines inserted into the wikimedia XML dump, identified the problem as the new use of the EoL bespoke FileIterator() code, which strips newlines. These are then put back on in WikimediaHarvester(), which results in a newline being erroneously added at the end of each part_XXX file. We cannot correct this in the WikimediaHarvester code, because if we delete the last newline of the file, we might be accidentally deleting a newline that happened to be present just before the split. The correct way to do this is to pass an option to FileIterator to keep newlines, which is implemented here.

Any other code which uses FileIterator to iterate through large files split into several pieces (? wikiPEDIA?) should also make use of this new parameter.

This pull request also makes a tweak to the make_valid_pagetitle() code to keep unicode formatting characters in titles, as seen e.g. in https://commons.wikimedia.org/wiki/File:Môr-wiail_Byseddog_%C2%96_Laminaria_digitata_-_Kelp_-_geograph.org.uk_-_692907.jpg