mpi-sws-rse/datablox — Issue #22: Need a way to call categorizers directly from another block
Author: jfischer
Created: 2012-01-08 21:04:32+00:00
State: open
Labels: Design

The categorizer as it is currently implemented won't work for my file analytics scenario. The problem is that it is called after the directory source block has already retrieved the file. I want to make the decision about whether to retrieve the file within Dir-Src, after it has read the file metadata but before it sends the log on the downstream blocks. This will avoid having to read and send large files that are not indexable.

I think the solution for this is to somehow make the Dir-Src block parameterized by a categorizer (function or block). There's a couple approaches we can take, depending on how general we want to be:

1. We could just have the Dir-Src block take an optional configuration parameter naming a python class to instantiate. This class would be called on each record before loading the file. Not sure how we ensure that the Python code to implement the Python class gets installed on the host running the block.
2. We could generalize this by having an inline categorizer/filter base class and letting the user specify a list of these in the configuration file. There would be a mapping from categorizers/filters to Engage resources to enable the framework to install the code on the host running the block.
3. The categorizer could just be a full-fledged block (like the existing categorizers) with some way to instantiate it inline.