ooici/coi-services — pull request #131 by dstuebe (2012-05-30 20:07:55+00:00)
Title: Search and nav

One rebased commit containing all of Lukes changes while implementing the Index, Catalog and Discovery Service.

Luke: IMS create

Luke: more stuff impl

Luke: Started on Unit Tests for Index Management

Luke: Bug in IMS, started Int test

Luke: added tests for delete and read

Luke: Placeholder

Luke: bump

Luke: Bump for resources

Luke: skeleton for catalog management

Luke: more work on CMS

Luke: Refactoring and impl

Luke: Added test for CMS

Luke: Unit tests for CMS

Luke: added span_resources

Luke: Add support for spanning resources

- span_resources in IMS will iterate through all known resource types
and create independent index 'types' in ElasticSearch based on known
resources.
- _map_resources will iterate through the resource schemas and create a
best-guess appropriate mapping. More features will follow-

TODO: Need to discuss further support for good indexing types such as a
date type in the resource which we can search and index with, as well as
a concrete definitionf or our geo-points. Complex types will need a
javascript filter and will be slow to initially index (not too much of a
problem).

Luke: Fix bugs with mappings

Luke: Feature: index and filter individual documents based on doc_id

- _individual_index now dynamically creates and updates a CouchDB
changes filter to reflect any number of desired document ids, great
for custom rivers and indexes.

Luke: Add features to IMS

- river_start allows the river to be started at a clients request
- index_resources allows a client to specifically designate which
resources to index from CouchDB
- span_resources was tweaked

Luke: Fixes support for individual document indexing

Luke: Unit tests for IMS

Luke: updated unit tests

Luke: adds mapped_fields

Luke: Unit tests for map_resources

Luke: Adds support for mapping resource fields

Luke: Added create_mapping and support for basic queries

Luke: Unit test for create_mapping

Luke: Unit tests for query

Luke: simple int test for query

Luke: Added server host/port config opts

Luke: Adds bootstrap

Luke: pull on upstream/master

Luke: adds bootstrap and resource_mapping

Luke: docstrings for everything

Luke: changed span resources fixed unit tests

Luke: in progress

Luke: Added support methods to CMS and IMS

- Deletions in IMS will accurately propagate through the associated
catalogs
- Catalogs properly aggregate across changes in indexes

Luke: Chanes the bootstrap for indexes

Luke: Error handling, unit tests and find_indexes

- Elasticpy 0.7 uses Requests instead of urllib2, error handling is much
better and wrapped so not to crash the container.
- Unit tests and INT tests updated to reflect changes in impl
- find_indexes implemented to have basic capability
- list_indexes now returns a dict with name,resource_id key value pairs.

Luke: Adds gevent compatible timeout handling in ES calls

- Wraps ES calls in an AsyncResult pattern with a timeout
- Timeouts are now pyon.core.exceptions.Timeout (doesn't crash
container)

Luke: Adds search_fields and catalog_fields to CMS

Luke: More support

- Adds support for intelligent resource filters
- Adds object deserialization for ES Queries

Luke: Fixes tests in support of new resource_mapping

Luke: Adds CRUD for views

IMS Refactoring

Luke: Adds index bootstrap service

Luke: Adds unit testing for index_bootstrap

Luke: Touches up on index_bootstrap

Luke: Changes create_catalog

Merge extern/ion-definitions

Luke: Removed excess logging

Luke: Merge extern/ion-definitions

Luke: Adds integration tests for CMS

Luke: Added one more index to std_indexes

Luke: adds initial support for couchdb index and bootstrapping

Luke: Adds collections

Luke: adds list_collections

Luke: CRUD for views

Luke: merge ion-defs

Luke: CRUD on views

Luke: starting on searches

Luke: Primitive Searching

Luke: Merge ion-definitions

Luke: Searching against catalogs and indexes

Luke: Unit tests for discovery query

Luke: IonObject Deserialization in Queries

Luke: Further support for indexes and searching

Luke: adds query against CouchDB indexes

Luke: support for primitive searching against couchdb indexes

Luke: Merge ion-definitions

Luke: ims integration tests

Luke: Added integration tests for index bootstrap

Luke: collection integration tests

Luke: Adds features and fixes

- BFS Traversal across associations
- CMS and IMS fixes for the new index resource types
- Added IonObject Support to IMS.list_indexes()

Luke: Fixes for tests

Luke: merge ion-definitions

Luke: Adds initial query DSL

Luke: merged extern/ion-definitions

Luke; query language definition and unit test

Luke: Adds more thorough unit testing of query dsl

Luke: First query

Luke: Adds tier-2 searching

Luke: Adds unit tests, futher implementation

Luke: adds catalog querying (simple)

Luke; adds rel for bootstrapping indexes (temp)

Luke: added support to search against views

Luke: support for querying against collections

Luke: Cleans up service definition

Luke: adds event notification for buffer exceeded

Luke: refactors some repetitive code

Luke: Manages RETIRED lcstate for cached documents

Luke: bump-ion defs

Luke: Adds bulk requests, unit tests

Luke: refactored some repetitive code

Luke: unit tests

Luke: limits and order

Luke: adds offsets to query

Luke: Bump ion-defs

Luke: fixes bug for empty results

- Prior to this bugfix, empty results when queried against a view or
catalog raised a BadRequest because of improper checking of empty
arrays vs None

Luke: adds features and integration tests

- sys_name now prefixes index names
- integration tests

Luke: adds more

Luke: refactors query language tests

Luke: refactoring

- Changed the relationship between views and catalogs
- Refactored some test code into a seperate file

Luke: fixes unit tests in CMS

Luke: searching by index name supported

- Search queries can now be against an index canonical name or a
resource's name.

Luke: adds integration test for data_product searching

- Meets L4-CI-SA-RQ-306

Luke: starts adding cluster configs

Luke: Cluster configurations, test cleanups

- Added more rigid support and fine grained tuning for index clustering
- Added seperate configurations for the _river index which controls the
HTTP connections to CouchDB
- added configurations to pyon.yml for these fields
- Integration tests for Discovery now do a good cleanup after the test
- Optimized shards/replicas in Discovery Integration Test for
performance

Luke: better int test cleanups

Luke: adds iterative traversal to discovery

Luke: adds support in search language

- Depth limited searches across associations can now be performed using
the Query DSL

Luke: adds find_association_mult to resource_registry

Luke: updates int tests for rr

Luke: changes association queries

- Adds better support for bulk queries against CouchDB for association
traversal, makes (lg n) requests instead of (n) requests for the
number of nodes.

Luke: adds install script for ES

Luke: adds ES support to README

Luke: readme changes and bump version on elasticpy

Luke: patches ingestion management

- Adds bounds checking for list bug

Luke: same for data_retriever

Luke: typo

Luke: Checks for ElasticSearch support

- New field in CFG defined in pyon.yml
- system.elasticsearch (Defaults to False) defines elasticsearch support
- Discovery raises a BadRequest for any query which uses ES
- Integration Tests are now skipped if ElasticSearch support is not
defined or enabled

Luke: uses global var in lieu of instance var

Luke: changes to setup.py

Luke: getting ready for push

Luke: elasticpy version

Luke: adds checks to bootstrap test for es