nathanmarz/storm — Issue #118: API redesign
Author: nathanmarz
Created: 2012-01-29 01:49:32+00:00
State: open

This issue lists ideas for redesigning the core API of Storm. The intention of this issue is to mark down ideas so they're not forgotten.

A redesign should allow for greater composability when developing higher level abstractions on top of Storm. Most users never hit these issues, because they don't do things that involved. However, the DRPC and Transactional topology abstractions "leak" a little bit (the id for the batch needs to be maintained in the first field of the tuple manually by the user), and those leaks can be completely plugged with more sophisticated primitives in Storm.

- a message should be able to be any type (list, map, object, whatever)
- arbitrary metadata for streams beyond "Fields". For example, you may not need any metadata, or maybe you have a different kind of metadata for your particular datatype
- different types of streams should work with different groupings. certain groupings like shufflegrouping, nonegrouping, globalgrouping are generic. but fieldsgrouipng is specific to a particular data model.
- should be able to layer messages on top of each other, like how network packets work. for example, coordinatedbolt stacks information about the coordination on top of whatever data the delegate bolt emits. similarly, the coordinatedbolt metadata stacks over the metadata for the delegate bolt. then when stream groupings are done, the "CoordinatedGrouper" can delegate to the grouping chosen by the user.
- these changes prevent the need to do things like choose a unique name for the first field of tuples like TransactionalSpout implementations need to do. instead they can wrap the underlying data in their own data model
- the subscription api should look like this: ".subscribe(component, stream, new TypeOfGrouping())"
- e.g. ".subscribe(component, stream, new CoordinatedGrouping(new FieldsGrouping("word"))
- the higher level abstraction can cleanly hide the nested groupings from you