pediapress/mwlib — pull request #14 by UltraNurd (2012-03-26 21:10:56+00:00)
Title: Allow mw-buildcdb to ignore redirects

I added an --ignore-redirects flag to mw-buildcdb, exposing the ignore_redirects __init__ parameter in DumpParser. This allowed me to build a CDB from a dump file, so that when I iterate over all articles I only obtain full pages, not a duplicate of each page under each redirect title (which was resulting in my code doing a lot of unnecessary processing, and throwing off the stats I was trying to accumulate).

Another option would be to somehow flag whether or not a page was a redirect when building the CDB, which you could then check when iterating over the CDB, but this seemed to be the best way to address the issue I was running into that I posted about on the Google Group: http://groups.google.com/group/mwlib/browse_thread/thread/b22682b8da9563a5