/dev/docs/data
Type: /type/page
body: value: ## Step 1: Get the data

The data we have and the data we want is all listed on the <a href="/about/help">how you can help</a> page. We've got a fair amount of it right now awaiting processing.

Here's where it is on <a href="ia">our servers</a> -- _note:_ this doesn't include other data listed in the how you can help page:

* Powells crawl: `wiki-beta:/1/pharos/crawl/powells` (aaronsw, phr)
* OCA full text: _fill in_ (phr)
* Indian Million Books Project: _in progress_ (anand)
* random Princeton University Press files: `wiki-beta:/1/princeton` (aaronsw; wget mirror)
* random Biblioteque National France files: `wiki-beta:/1/ttk/bnf` (bill)
* book covers: `wiki-beta:/1/pharos/covers` and `wiki-beta:/2/pharos/bookcovers` (aaronsw)
* bookmooch data: `wiki-beta:/1/pharos/covers/bookmooch` (aaronsw)
* librarything data: `wiki-beta:/1/pharos/covers/librarything` (aaronsw)
* librarything data: `wiki-beta:/1/pharos/crawl/librarything` (aaronsw; wget mirror)
* [fred20 data](http://www.ibiblio.org/fred2.0/authorities/): `wiki-beta:/1/pharos/crawl/fred20` (aaronsw)
* Access Copyright data: `wiki-beta:/1/pharos/onix/accesscopyright` (aaronsw, rejon)
* ONIX data: `wiki-beta:/1/pharos/onix/originals` (aaronsw, alexis)
* recently-uploaded ONIX data: `wiki-beta:/tmp/incoming` (aaronsw, alexis)
* [[/dev/docs/data/PSU|PSU data]]: `wiki-beta:/1/pharos/onix/psu` (aaronsw)
* Stanford copyright data: `wiki-beta:/1/pharos/onix/copyright` (aaronsw)
* bulk.resource.org data: `wiki-beta:/2/pharos/crawl/bulk.resource.org` (aaronsw; wget mirror)
* random Powells data: `wiki-beta:/2/pharos/crawl/ftp.powells.com` (aaronsw; wget mirror)
* partial isbndb data:  `wiki-beta:/2/pharos/crawl/isbndb.com` (aaronsw; wget mirror)
* onlinebooks project data: `wiki-beta:/2/pharos/crawl/onlinebooks.library.upenn.edu` (aaronsw; wget mirror)
* random book text:  `wiki-beta:/2/pharos/crawl/www.??????.us` (aaronsw; wget mirror)
* ISBNs: `wiki-beta:/2/pharos/isbns` and more recently `apollonius:/0/pharos/crawl/more_isbn_project` (aaronsw; workspace)
* worldlibrary.net: `wiki-beta:/2/pharos/pdfgrab` (aaronsw; wget mirror)
* wikipedia dumps: `wiki-beta:/2/wikipedia` (aaronsw; wget mirror)
* wikipedia dumps: `apollonius:/2/feisty/x-home/phr/wiki` (phr)
* dbpedia dumps: `zenodotus:/2/pharos/crawl/dbpedia/` (aaronsw; wget mirror)
* wikia dumps: `zenodotus:/2/pharos/crawl/wikia/` (aaronsw; wget mirror)
* amzn data: `apollonius:/3/aaronsw/isbn` and `wiki-beta:/2/pharos/isbns/crawl` (aaronsw)
* MBL/WHOI catalog: `wiki-beta:/1/pharos/marc` (aaronsw)
* enron emails: `apollonius:/3/aaronsw/crawl`(aaronsw; wget)
* elsevier covers: `wiki-beta:/1/pharos/onix/elsevier` (aaronsw; wget)
* LC additional data: `apollonius:/0/pharos/crawl/lc_catdir` (aaronsw; wget)
* undemocracy: `zenodotus:/2/pharos/crawl/dbpedia/undemocracy` (aaronsw; wget mirror)
* bowker: `apollonius:/3/aaronsw/crawl/bowker` (aaronsw)
* nyt front pages: `apollonius:/0/pharos/crawl/nytimes_front_page` (aaronsw; wget mirror)
* drexel catalog: `apollonius:/0/pharos/crawl/drexel` (aaronsw; wget)
* laurentian catalog: `apollonius:/3/aaronsw/crawl/laurentian` (aaronsw; wget)
* orbiscascade identifier mappings: `wiki-beta:/2/pharos/crawl/orbiscascade` (aaronsw; wget; hasn't been used on amzn yet)

Here's stuff that's been archived in the petabox:

* [unc catalog](http://www.archive.org/details/unc_catalog_marc/): `apollonius:/0/pharos/crawl/unc` (aaronsw)
* [mit catalog](http://www.archive.org/details/barton_catalog_mods): `apollonius:/0/pharos/crawl/mit` (aaronsw)
* [umich records](http://www.archive.org/details/umich_pdscan_records): `apollonius:/0/pharos/crawl/umich_pdscan_records` (aaronsw; wget)
* [amazon similarity graph](http://www.archive.org/details/amazon_similarity_graph/): `apollonius:/0/pharos/crawl/amazon_similarity_graph` (aaronsw; [awscrawl.py](http://theinfo.org/get/code/awscrawl.py))
* [oregon catalogs](http://www.archive.org/details/marc_oregon_summit_records): `apollonious:/3/aaronsw/crawl/oregon/marc_oregon_summit_records` (aaronsw; flash drive)
* [uoft catalog](http://www.archive.org/details/marc_university_of_toronto): `wiki-beta:/2/pharos/crawl/uoft` (aaronsw; wget)
* [muohio catalog](http://www.archive.org/details/marc_miami_univ_ohio/): `apollonius:/3/aaronsw/crawl/muohio` (aaronsw; wget)
* [wwu catalog](http://www.archive.org/details/marc_western_washington_univ): `apollonius:/3/aaronsw/crawl/wwu/marc_western_washington_univ` (aaronsw; cdrom)
* [boston college catalog](http://www.archive.org/details/marc_boston_college): `wiki-beta:/2/pharos/crawl/bostoncollege/` (aaronsw; ftp)

### in progress

* lc content (incl. books): `apollonius:/1/pharos/crawl/lc` (aaronsw; wget mirror)
* stanford books: `apollonius:/0/pharos/crawl/stanford` (aaronsw; in progress)

## Step 2: Process it

Aside from some special cases (e.g. lists of ISBNs, book covers, holdings data), we take each data sources, write a processor for it, and output Python dictionaries.

Right now the code we have for this is in our repository in the `catalog` directory. There is code to process a number of record formats, including MARC and ONIX. 

## Step 3: Merge it

As records are added, an algorithm detects whether the book is already represented in the database. In that case, some new fields from the incoming record may be added to the record in the database, such as additional identifiers, new subjects, and tables of contents. The success of determining duplicates depends on the quality and accuracy of the data in the records. We hope to make it easy to merge duplicates manually through the user interface so that Open Library users can do what the algorithm cannot.

## Step 4: FRBRize it

Then we need to go through and detect relationships between works (example: all of these editions of Tom Sawyer are all editions the same conceptual work). From this we can add relationships to each object and create new objects (like works).

This is not implemented yet.

## Step 5: Import it

Once we have all this, we need to start giving it identifiers and importing it into ThingDB.

Right now there is some simple code for this in the `catalog` directory with some silly identifier schemes.
links: ia
/dev/docs/data/PSU
/about/help
title: The State of Data Import
m: edit