Package: featureflow
featureflowfeatureflow is a python library that allows users to build feature
extraction pipelines in a declarative way, and control how and where
those features are persisted.UsageThe following example will compute word frequency in individual text
documents, and then over the entire corpus of documents, but featureflow
isn’t limited to text data. It’s designed to work well with
sequential/streaming data (e.g. audio or video) that is often processed
iteratively, in small chunks.You can seeall the code in this example in one place
here.We can define a graph of processing nodes like this:importfeatureflowasff@ff.simple_in_memory_settingsclassDocument(ff.BaseModel):"""
    Define the processing graph needed to extract document-level features,
    whether, and how those features should be persisted.
    """raw=ff.ByteStreamFeature(ff.ByteStream,chunksize=128,store=True)checksum=ff.JSONFeature(CheckSum,needs=raw,store=True)tokens=ff.Feature(Tokenizer,needs=raw,store=False)counts=ff.JSONFeature(WordCount,needs=tokens,store=True)We can define the individual processing “nodes” referenced in the graph
above like this:importfeatureflowasfffromcollectionsimportCounterimportreimporthashlibclassTokenizer(ff.Node):"""
    Tokenize a stream of text into individual, normalized (lowercase)
    words/tokens
    """def__init__(self,needs=None):super(Tokenizer,self).__init__(needs=needs)self._cache=''self._pattern=re.compile('(?P<word>[a-zA-Z]+)\W+')def_enqueue(self,data,pusher):self._cache+=data.decode()def_dequeue(self):matches=list(self._pattern.finditer(self._cache))ifnotmatches:raiseff.NotEnoughData()last_boundary=matches[-1].end()self._cache=self._cache[last_boundary:]returnmatchesdef_process(self,data):yieldmap(lambdax:x.groupdict()['word'].lower(),data)classWordCount(ff.Aggregator,ff.Node):"""
    Keep track of token frequency
    """def__init__(self,needs=None):super(WordCount,self).__init__(needs=needs)self._cache=Counter()def_enqueue(self,data,pusher):self._cache.update(data)classCheckSum(ff.Aggregator,ff.Node):"""
    Compute the checksum of a text stream
    """def__init__(self,needs=None):super(CheckSum,self).__init__(needs=needs)self._cache=hashlib.sha256()def_enqueue(self,data,pusher):self._cache.update(data)def_process(self,data):yielddata.hexdigest()We can also define a graph that will process an entire corpus of stored
document features:importfeatureflowasff@ff.simple_in_memory_settingsclassCorpus(ff.BaseModel):"""
    Define the processing graph needed to extract corpus-level features,
    whether, and how those features should be persisted.
    """docs=ff.Feature(lambdadoc_cls:(doc.countsfordocindoc_cls),store=False)total_counts=ff.JSONFeature(WordCount,needs=docs,store=True)Finally, we can execute these processing graphs and access the stored
features like this:from__future__importprint_functionimportargparsedefprocess_urls(urls):forurlinurls:Document.process(raw=url)defsummarize_document(doc):return'doc{_id}with checksum{cs}contains "the"{n}times'.format(_id=doc._id,cs=doc.checksum,n=doc.counts.get('the',0))defprocess_corpus(document_cls):corpus_id=Corpus.process(docs=document_cls)returnCorpus(corpus_id)defsummarize_corpus(corpus):return'The entire text corpus contains "the"{n}times'.format(n=corpus.total_counts.get("the",0))if__name__=='__main__':parser=argparse.ArgumentParser()parser.add_argument('--url',help='specify one or more urls of text files to ingest',required=True,action='append')args=parser.parse_args()process_urls(args.url)fordocinDocument:print(summarize_document(doc))corpus=process_corpus(Document)print(summarize_corpus(corpus))To see this in action we can:pythonwordcount.py\--urlhttp://textfiles.com/food/1st_aid.txt\--urlhttp://textfiles.com/food/antibiot.txt\...InstallationPython headers are required. You can install by running:apt-getinstallpython-devNumpy is optional. If you’d like to use it, theAnacondadistribution is highly
recommended.Finally, justpipinstallfeatureflow