ComputationHandler and GeospaceHandler

This change adds the concept of a ComputationHandler, designed to perform computations for computed columns from one or more source columns during ingress / upserts.   It also adds a handler for Geospace / geo region coding of point columns that are ingressed.
You mean return an Iterator[SoQLType]?   We can do that, but we still have
to pass in the ColumnRecord, because the computation needs the parameters.
  More considerations:

- If we chain ComputationHandlers, it's very easy if the input and output
types are the same, you can just do
Handler1.compute(Handler2.compute(source, cr), cr) ....
 (though I suppose having something else stitch the new column in isn't
hard either)
- Iterators are stateful.   If you give an iterator to the computation
handler, then it has already gone to the end of the rows, and after you get
the SoQL iterator back, you won't be able to go back through the original
rows to zip up the data
- When we add concurrency, many threads will be doing this computation.
Not necessarily an argument against.


On Wed, Jun 18, 2014 at 4:24 PM, urmilan <notifications@github.com> wrote:

> In
> soda-fountain-lib/src/main/scala/com/socrata/soda/server/highlevel/ComputationHandler.scala:
>
> > +trait ComputationHandler {
> > +  // The type of computation this handler handles.  Should be unique.
> > +  val computationType: String
> > +
> > +  /**
> > +   * Handles the actual computation.  Should be lazy if possible, otherwise will introduce
> > +   * significant latency.  IE, try not to convert the Iterator to a regular Seq or Array.
> > +   *
> > +   * @param sourceIt an Iterator of source rows, each row is a JValue (actually an JObject
> > +   *                 of key-value pairs, where the key is the column name).  Anything other
> > +   *                 than a JObject is not for upserts and can be ignored.
> > +   * @param column a ColumnRecord describing the computation and parameters
> > +   * @return an Iterator[JValue] for the output rows.  The JValue must be a JObject containing
> > +   *        a key matching column.name with the computed value.
> > +   */
> > +  def compute(sourceIt: Iterator[JValue], column: MinimalColumnRecord): Iterator[JValue]
>
> This comment is in the context that we are going to change sourceIt to
> Iterator[Map[String, SoQLValue]] : can we also make this method just return
> a SoQL type? I don't think the computation handler should have to know the
> name of the column it's computing and be responsible for adding the new
> SoQLType to the map. This shold be external to ComputationHandler.
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/socrata/soda-fountain/pull/5/files#r13946328>.
>



-- 
Evan Chan   |   Principal Engineer, Socrata     |    @evanfchan
*Socrata <http://www.socrata.com/> *| The Data Experience Company™
*What does Socrata do? <https://www.youtube.com/watch?v=yyz86MlkriQ> *(watch
our 2 min video)