Project: springxd
Issue XD-2232: Add support for estimating the median value of a data stream.
Often it is useful to have access to the median value for fields of a data stream since they are more robust with respect to outliers. The median is defined as the value of a dataset such that, when sorted, 50% of the data is smaller than the value and 50% of the data is larger then the value. Ordinarily this is difficult to calculate on a stream because it requires the collection and sorting of all data. The median of a data stream can be approximated with a technique called stochastic averaging. To approximate the median value of a data stream one could use the following approach: Given the current estimate of the median {{M}}. If the next observed value in the stream is larger than {{M}}, increase the current estimate by {{r}} (= the learning rate). If it is smaller, decrease the estimate by {{r}}. When {{M}} is close to the median, it increases as often as it decreases, and therefore it stabilizes. The following example shows a primitive implementation of the above mentioned algorithm in groovy (to be placed under ${XD_HOME}/modules/processor/scripts). {code:groovy} import org.springframework.xd.tuple.TupleBuilder; import org.springframework.xd.tuple.Tuple; /** * Stochastic averaging to compute the median of a data stream. * To approximate this value using stochastic optimization, the value * of interest is the current estimate of the median M. If the next observed * value of the stream is larger than M, increase by r. If it is smaller decrease the * estimate by r. When M is close to the median, it increases as often as it decreases, * and therefore stablilizes. * r denotes the learningRate. */ enum MedianEstimator{ INSTANCE //TODO Add support for estimating multiple medians /* * The current median value. */ double current = Double.POSITIVE_INFINITY public double update(double value, double learningRate){ //Initialize current with given value if(current == Double.POSITIVE_INFINITY){ current = value } if(current == value){ return current } //Move current value towards the median current = current + (current < value ? learningRate : -learningRate) return current } } //TODO Make learning rate configurable def learningRate = 0.7 //TODO Add support for dynamic field selection. double median = MedianEstimator.INSTANCE.update(payload.getDouble('value'), learningRate) def fieldNames = new ArrayList(payload.getFieldNames()) def fieldValues = new ArrayList
Story points: 5