Patent ID: 7447801
Filing Date: 2008-11-04
Classification: G06F,H04L

Abstract:
1. A method for managing a data flow, the method comprising: composing a serial arrangement of at least first, second, and third stream components as a pipeline for incrementally transferring a sequential dataset from a source to at least one destination; using the first stream component in the serial arrangement to sequentially route increments of the dataset from the source to the second stream component; using the second stream component in the serial arrangement to sequentially buffer each of the increments of the dataset received from the first stream component in a buffer, and to route each of the buffered increments of the dataset to the third stream component before a subsequent increment of the dataset is received from the first stream component to be buffered; and using the third stream component in the serial arrangement to sequentially perform a data transformation on each of the increments of the dataset received from the second stream component, and to route each of the transformed increments of the dataset to the at least one destination before a subsequent increment of the dataset is received from the second stream component to be transformed, wherein the size of each of the increments of the dataset routed by the first stream component corresponds to the size of the buffer in order to limit memory usage of the serial arrangement to the size of the buffer for each data increment, and wherein each of the first, second, and third stream components includes a stream wrapper encapsulating a stream module, the stream wrapper instructing the stream module to perform a read or write operation on a data file for an amount of data corresponding to the size of the buffer.