Squashed commit of the following:

commit 61c2738a6083c2e0b0203b297f046c4a7b01f8b7
Author: David Goehrig <dave@dloh.org>
Date:   Mon Oct 28 13:08:57 2013 +0000

    Adding readme

commit 4931d130e2db2187df538a0af6ac16fd2849a33a
Author: David Goehrig <dave@dloh.org>
Date:   Wed Jun 5 03:53:23 2013 +0000

    Adding some test data

    This is used for testing the index

commit 0c19f7852ac73706a426dfc137dc7374c45428f9
Author: David Goehrig <dave@dloh.org>
Date:   Wed Jun 5 03:10:51 2013 +0000

    Adding a utility function to convert bins to terms

    When using Elixir, the typical representation of a term is a
    "" string which is an erlang binary, providing automatic
    conversion makes:

    :index_server.search(:index_server_testuser_testindex, "test")

    actuall work

commit d8d8d32dc5f6eb434c4d5ad6a8a3cea809e6e6a2
Author: David Goehrig <dave@dloh.org>
Date:   Wed Jun 5 02:54:49 2013 +0000

    Adding a makefile to ease dev

    compile - compiles
    erl - runs erl in local dir with correct paths
    iex - runs iex in the local dir with erl paths set

commit ffd2110648eb47e89dbf05e9e54b0b6f51a8b6e9
Author: David Goehrig <dave@dloh.org>
Date:   Wed Jun 5 02:44:46 2013 +0000

    Index server now has a proper test.

    It now queries the searchapi project and indexes the /test page.
    If you modify the test page, you need to modify the test.

    There is now a index_server api for searching and indexing files:

    index(Index,Url).
    search(Index,Term).

    Index is the Pid or name of a index_server instance, and the other is
    a list.

    TODO add -spec so dialyzer can introspect about types, etc.

commit 5928e606be8aeb3f764ba23c07a0bf8779763441
Author: David Goehrig <dave@dloh.org>
Date:   Wed Jun 5 02:11:54 2013 +0000

    updating index server to index and report

commit 1807d99a905ad369c6df411c5fe193e781c4e19c
Author: David Goehrig <dave@dloh.org>
Date:   Mon May 27 17:29:21 2013 +0000

    ranksort is returning high to low

commit ad3f31bacf978ad5f24292d1238dc5d11f03dde3
Author: David Goehrig <dave@dloh.org>
Date:   Mon May 27 17:14:12 2013 +0000

    Adding archive support

    archive.erl adds support for zip files containing per-user indexes

    document.erl now has support for searching for a list of urls matching a
    term

    index.erl now supports rank sorting based on term frequency (need IDF
    factor in the future)

    index_server is a basic stub for a future index service

    server.erl is a template for gen_server code because I didn't make a
    macro yet.

commit 75d42b5b73b6b11f40a7496d490a1e6d6bf2c41c
Author: David Goehrig <dave@dloh.org>
Date:   Mon May 27 15:46:16 2013 +0000

    Updating a few more bits to interface with the new bin format

commit 3784a733f791e1b73c6a3a365720c62ffe887102
Author: David Goehrig <dave@dloh.org>
Date:   Sun May 19 12:03:50 2013 +0000

    peek / poke and partial encode/decodes

    This is a massive speed improvement eve over the original.
    By not mutating most of the binaries and doing prepend only,
    was able to get indexing full site from 90s down to 6s.

commit 4c927eda6d5ebfcb1cc1e6523a10dfc5d2db2ebe
Author: David Goehrig <dave@dloh.org>
Date:   Sun May 19 05:30:09 2013 +0000

    varint variation compresses full index to under 1M

commit 9b241383ab039df9eae1c19afb958c45862049ce
Author: David Goehrig <dave@dloh.org>
Date:   Sun May 19 05:05:36 2013 +0000

    working test case for the list based index

commit 43766e4b10fecc5cc744b63b8f2ef30d79dd2785
Author: David Goehrig <dave@dloh.org>
Date:   Sun May 19 04:19:55 2013 +0000

    convert the doc tule to a list

commit 17cb664154c711385ee129217c9db6ab57110f03
Author: David Goehrig <dave@dloh.org>
Date:   Sun May 19 04:09:47 2013 +0000

    varint done, removing freq

    plan is to move doc term count to doc index
    word counts and offsets tomalso become a list
    use varint to encode/decode on the fly to save space

commit dea8ecbe0c5236f1abfb735ca4d686b6eebf6a7c
Author: David Goehrig <dave@dloh.org>
Date:   Sun May 19 03:16:01 2013 +0000

    adding variable byte encoding for integers

    we can compress lists of integers far more efficiently using a
    variable encoding  this comes at a runtime decode cost, but
    with sufficiently large lists the space savings is enormous

commit 26c46c353a1333fac7213df068885a00fa099031
Author: David Goehrig <dave@dloh.org>
Date:   Wed May 15 04:10:08 2013 +0000

    Adding the secondary index

    This is the secondary index which adds just a few k of additional
    indexing but reduces the primary index by another 3MB

commit 9a3da622517e714cb83ec56db2d3d4a421fdc6ce
Author: David Goehrig <dave@dloh.org>
Date:   Wed May 15 04:09:01 2013 +0000

    Moved document URLs out of the index file

    Added a new secondary index and reduced the index size to 3.9MB

commit 8a401a83981e68a35c5d629f3471bd93d5b8c3c6
Author: David Goehrig <dave@dloh.org>
Date:   Wed May 15 03:19:53 2013 +0000

    removing debug data

commit 2cc583dfe64c1b22995e5eb3646e33bba1ee326a
Author: David Goehrig <dave@dloh.org>
Date:   Wed May 15 03:08:29 2013 +0000

    html stripping knocked off 2MB from index for blog