Project: usergrid
Issue USERGRID-324: [SPIKE] Prototype a few distributed realtime parallel processing systems
We need a system to allow us to build distributed/parallel data process flows. Ultimately, this system must have the following requirements at a component level. h1. Software Requirements # Work well within Reactive/Streams logic # Allow easy operationalization. No additional external systems should be required # Allow failover during node failure, and automatic recovery. Processing should not stop/restart because of a failure, it should resume. # Not additional state administration should need to be built. Distributed futures should be an inherent part of the framework. Some examples are the following. # Migrations # Import/Export # Distributed Indexing on heavily connected entities # Post processing deletes # Collection deletes # Application deletes I have the following requirements. # You can define a deployment topology and limit the number of sub processes/workers in the workflow # Ability to reject requests when there is no capacity # Preferably, do not introduce another dependency (like Zookeeper) and deploy it in the stack war file # An easy intuitive interface for programming flows which will work in a single node, or clustered environment # Administrative status, giving the ability to poll operational status via REST for automated monitoring h2. Examples h3. Guaranteed post processing # A write occurs. It is written to Cassandra and a post processing message is triggered. ## This message must be persistent, and allow for recovery during complete failure. ## If there is no capacity for post processing in the system, we should reject this request via back pressure ## The post processing should return a future (or similar) that could be blocked on if the caller chooses to. # The HTTP call is returned to the user # Post process indexing occurs asynchronously h3. Reindex all entities in the system. # Launch a root process. This process emits all application ids within the system. # Child processes receive the application id. For each app, create the index in elastic search, then emit all collections. # Child processes receive the collections and app ids. For each collection, emit the entity id # Child process receives the app, collection, and id. For each entity, get it's edges, and re-index the documents within elastic search. h3. Delete a collection Realtime -> Update collection alias to point to a new internal collection name. Fire delete collection task. Job process # Launch root process. Load previous collection name and emit to 2 child tasks. # Child task 1: Remove every entity of the previous type from elastic search using bulk delete until empty. # Child task 2: Iterate every entity and remove it from Cassandra, as well as it's graph edge.
Story points: 8