internetarchive/openlibrary — Issue #131: Create a dump of ia metaxml dump regularly
Author: anandology
Created: 2012-01-18 11:29:27+00:00
State: open

Create a dump of ia collections info from metaxml for all the archive.org items referenced in Open Library.

The dump should have 3 columns, ol-book-key, itemid, ia-collection-info-as-json.

I've generated on dump manually and it is available at:

http://www.archive.org/download/ol_dump_2011-12-31/ol_dump_iameta_2011-12-31.txt.gz

The metaxml of an item be downloaded by new metadata API. http://www.archive.org/metadata/opticksortreatisnewt

Edward has a script to download metaxml of all the items on a node.

http://ia700405.us.archive.org/~edward/ls_dir.php
gives you all the drives on a machine

this reads all the meta XML on that drive:
http://ia700405.us.archive.org/~edward/read_all_meta.php?dir=0