Package: datafun
🍻 datafunThe datafun allows for loading existing datasets or files with different formats and treat them as streaming pipelines. It also allows one to add operations like filter and map in a functional way executed lazily while loading data.OverviewArchitectureInstallUsageAvailable datasetsCustom DatasetsOverviewWith datafun you can:Load files locally from specific file formats, eg. CSV, JSON etc.Load files from remote sources, eg. a CSV or a JSON from a Google Cloud Storage bucket.Apply streaming transformations on the fly by applying functional operations (filter, map...) to it.Define data transformations to be stored and later used and extended by anyone. This will contribute to an internal hub of common datasets and pipelines used many times.ArchitectureA datafun Dataset is both a lazy stream of data and lazy sequential pipeline of operations.It is a stream of data: when you iterate over the dataset, one element at a time is generated and returned to the caller.It is a lazy sequential pipeline: every time an element needs to be returned, the pipeline of operations is executed on the current element.Node types.The DatasetSource class is the first node of the pipeline and is responsible for loading the data from a local or remote storage, then passing it to the next node in the pipeline. DatasetNode objects representing filter or map operations can be added to the pipeline.Communication.The pipeline communicates in a message-passing fashion. Stream objects are passed between nodes, representing the state of the stream and containing the current element. A stream may be a StartOfStream, EndOfStream, PullStream, or just Stream (when containing data). To give an example of the communication, consider the following pipeline:TextDataSource->Map(lambdax:x.lower())->Filter(lambdax:"messiah"inx))and the following stream of data:"Dune" -> "Dune Messiah" -> "Children of Dune" -> ...This is what happens when you begin iterating over the stream:StartOfStream triggers TextDataSource, which reads the first elementDuneStream is forwarded to Map, which transformsDuneintoduneStream is forwarded to Filter, the lambda inside fails the check, so a PullStream is sent backwardPullStream arrives to Map, which sends it back to TextDataSourceTextDataSource reads the PullStream, generates another element,Dune Messiah, then issues a Stream forward....This Stream arrives at the end, somessiahit is returned to the caller....Available transformation operations.nameArgumentsDescriptionfilterf: Callable[T, bool]Filters elements according tof.mapf: Callable[T, S]Maps every element to another with functionf.flat_mapf: Callable[T, Iterable[S]] = lambda x: xA Flat Map takes a list and returns every element, one at a time. The given functionf, given element of timeT, must return an Iterable of typeS. If not provided,fis the identity.samplingp: float, seed: intSamples elements according to the sampling ratiop:0. < p <= 1.uniqueby: Callable[T, U]Removes duplicates.byargument specifies a function that extracts the desired informationaggregateinit: Callable[[], S]agg: Callable[[T, S], S]reduce: Callable[[S], R]Aggregates stream using functionf. Hint: given initial valueinitof typeSforaggregated stream, mapsfto stream (value:T,aggregated:S) wherevalueis anelement of the stream, andaggregatedis the current aggregate value.reduceis applied to the aggregated value, if providedzip*iterablesZips elements from multiple dataset, like python zip()joinother: Dataset, key: Callable, key_left: Callable, key_right: CallableJoins two datasets based on provided key functions that specify the path in the dictionary. Either specifykeyfor both, orkey_leftandkey_right.limitn: intLimits the number of elements in the streams to the firstncache/Caches result of previous nodes into memory. Useful when the pipeline is executed multiple times. Be careful with memory intensive operations.repeat/Make the dataset infinite. Be careful that this does NOT cache the loading of data. Usa cache() beforehand if needed.Datasets overloads basic python operations:+,-,*,/.You can see examples for every operation in thededicated notebook.Available info operations.nameArgumentsDescriptioninfo/Prints dataset descriptionsummary/Prints list of transformations in the pipelineschema/Prints schema of dataset. Useful for csv, json, jsonlAvailable output operations.nameArgumentsReturn typeDescriptionshown: intstrPrint first n elements as stringtaken: intList[T]Returns the first n elements of the datasettake_whilef: Callable[..., bool]List[T]Returns all elements satisfying fcollect/List[T]Returns a list containing all elements. Also prints progress while collecting.Available other operations.nameArgumentsDescriptionclone/Deep clone datasetInstallpipinstalldatafunUsageimportdatafunasdfnLoad a raw dataset (local|remote)ds:Dataset=dfn.load('text',path="path/to/dune_frank_herbert.txt")forelinds:# do something with elLoad a dataset from iterable data structureds=dfn.load(range(N))ds.take(5)# take first 5 integers>>[0,1,2,3,4]Adding ops to the pipeline, evaluated lazily when generating datads=dfn.load('json',path="path/to/file.json")ds2=ds.filter(lambdax:x['KEY']>10)ds2=ds2.map(lambdax:x**2)print(ds2.summary())# Shows pipeline opsforelinds2:# do something with el.# This will execute the defined ops one element at a timeEasy streaming transformation of remote dataset!# Streaming normalize on the flyds=(dfn.load('gcs-jsonl',path='gs://my_bucket/timeseries.jsonl').map(lambdax:x['value']))mean=3.14# let's assume to have a meannorm_ds=ds-meanforvalueinnorm_ds:print(value)You can see examples for every operation in thededicated notebook.Available datasetsIDClassElem. typeDescriptiontextTextDatasetstrStreams the local text file(s) one line at a time. Glob supportedcsvCSVDatasetdictStreams the local CSV file(s) one row at a time. Glob supportedjsonJSONDatasetdictStreams the local JSON file(s) one at a time. Glob supportedjsonlJSONLinesDatasetdictStreams the local JSONLines file(s) one line at a time. Glob supportedgcsGCSDatasetstrDownload files from Google Cloud Storage, then streams paths to be loaded locally. Glob supportedgcs-textGCSTextDatasetstrAs gcs, but also opens the text file(s) and streams one line a time. Glob supportedgcs-csvGCSCSVDatasetdictAs gcs, but also opens the CSV file(s) and streams one row a time. Glob supportedgcs-jsonGCSJSONDatasetdictAs gcs, but also opens the JSON file(s) and streams it one at a time. Glob supportedgcs-jsonlGCSJSONLinesDatasetdictAs gcs, but also opens the JSONLines file(s) and streams one line at a time. Glob supportedelkELKDatasetdictConnects to ELK instance, runs json query specified inpath, then returns the resulting docs one at a timerestRESTDatasetdictStreams the response from a REST API.TextDatasetNameTypeRequiredDefaultDescriptionpathUnion[List[str]], str]YesThe local path to be loadedencodingstrNo'utf-8'The encoding to be used when opening the fileallowed_extensionsSequence[str]NoNoneThe supported extensions (others will be discarded). None means all extensionsReturned element type:strCSVDatasetNameTypeRequiredDefaultDescriptionpathUnion[List[str], str]YesThe local path to be loadedencodingstrNo'utf-8'The encoding to be used when opening the fileallowed_extensionsSequence[str]No('csv')The supported extensions (others will be discarded). None means all extensionsdelimiterstrNo','The column delimiterReturned element type:dict. Each element represent a row from the CSV(s). The keys are the CSV column names.JSONDatasetNameTypeRequiredDefaultDescriptionpathUnion[List[str], str]YesThe local path to be loadedencodingstrNo'utf-8'The encoding to be used when opening the fileallowed_extensionsSequence[str]No('json')The supported extensions (others will be discarded). None means all extensionsReturned element type:dict. This dictionary directly maps the JSON file.JSONLinesDatasetNameTypeRequiredDefaultDescriptionpathUnion[List[str], str]YesThe local path to be loadedencodingstrNo'utf-8'The encoding to be used when opening the fileallowed_extensionsSequence[str]No('json', 'jsonl')The supported extensions (others will be discarded). None means all extensionsReturned element type:dict. This dictionary maps a single JSON line from the file.GCSDatasetNameTypeRequiredDefaultDescriptionpathUnion[List[str], str]YesThe remote path to be loadeddownload_pathstrNo'$HOME/.cache/data-engineering'The path where the file(s) will be downloadedservice_accountstrNoNoneThe service account path to use. If missing will use the default user accountprojectstrNoNoneThe project id to use. If missing will use the user configured project itReturned element type:str. Each element represent a local path where to find the file downloaded from GCS.GCSTextDatasetNameTypeRequiredDefaultDescriptionpathUnion[List[str], str]YesThe remote path to be loadeddownload_pathstrNo'$HOME/.cache/data-engineering'The path where the file(s) will be downloadedservice_accountstrNoNoneThe service account path to use. If missing will use the default user accountprojectstrNoNoneThe project id to use. If missing will use the user configured project itencodingstrNo'utf-8'The encoding to be used when opening the fileReturned element type:str. Each element represent a line from the remote file.GCSCSVDatasetNameTypeRequiredDefaultDescriptionpathUnion[List[str], str]YesThe remote path to be loadeddownload_pathstrNo'$HOME/.cache/data-engineering'The path where the file(s) will be downloadedservice_accountstrNoNoneThe service account path to use. If missing will use the default user accountprojectstrNoNoneThe project id to use. If missing will use the user configured project itencodingstrNo'utf-8'The encoding to be used when opening the fileallowed_extensionsSequence[str]No('csv')The supported extensions (others will be discarded). None means all extensionsdelimiterstrNo','The column delimiterReturned element type:dict. Each element represent a row from the CSV(s). The keys are the CSV column names.GCSJSONDatasetNameTypeRequiredDefaultDescriptionpathUnion[List[str], str]YesThe remote path to be loadeddownload_pathstrNo'$HOME/.cache/data-engineering'The path where the file(s) will be downloadedservice_accountstrNoNoneThe service account path to use. If missing will use the default user accountprojectstrNoNoneThe project id to use. If missing will use the user configured project itencodingstrNo'utf-8'The encoding to be used when opening the fileallowed_extensionsSequence[str]No('json')The supported extensions (others will be discarded). None means all extensionsReturned element type:dict. Each element is a dict directly mapping a JSON file.GCSJSONLinesDatasetNameTypeRequiredDefaultDescriptionpathUnion[List[str], str]YesThe remote path to be loadeddownload_pathstrNo'$HOME/.cache/data-engineering'The path where the file(s) will be downloadedservice_accountstrNoNoneThe service account path to use. If missing will use the default user accountprojectstrNoNoneThe project id to use. If missing will use the user configured project itencodingstrNo'utf-8'The encoding to be used when opening the fileallowed_extensionsSequence[str]No('json', 'jsonl')The supported extensions (others will be discarded). None means all extensionsReturned element type:dict. Each element is a dict mapping a JSON line from the file(s).ELKDatasetNameTypeRequiredDefaultDescriptionpathUnion[List[str], str]YesThe local path to a query written in JSONhoststrYesElastic host URLportstrYesElastic host portusernamestrYesElastic authentication usernamepasswordstrYesElastic authentication passwordindexstrYesElastic index to querystart_isodatestr (ISO datetime)YesElastic start date range with format: "2021-09-15T10:00:00.000Z"end_isodatestr (ISO datetime)YesElastic end date range with format: "2021-09-15T10:00:00.000Z"date_fieldstrNo@timestampElastic date field. Can be nested into list, eg. "messages.date"date_field_separatorstrNo.Separator for date_field used to split the path. Use different ones to NOT split and consider date_field as single fieldReturned element type:dict. Each element is a document matching the given query.RESTDatasetNameTypeRequiredDefaultDescriptiondownload_pathstrNo'$HOME/.cache/data-engineering'Where to download dataheadersdictNo{}Call Headersauth_headersdictNo{}Authentication headersparamsOptional[dict]NoNoneURL key value paramsmethodstrNo'GET'Call method, 'GET', 'POST'...next_url_fCallable[[dict], Optional[str]]Nolambda x: NoneFunction to extract the next url to call from the json response for paginated APIs. Should return None to mark the end or the URL stringReturned element type:dictrepresenting the json response.Custom datasetsMethod 1. Functional styledefmy_load(path,**kwargs):ds=(dfn.load('json',path=path,**kwargs).filter(lambdax:x['KEY']>10).map(lambdax:x**2))returndsMethod 2. Extend DatasetWrite classes@dataclassclassELKDatasetConfig(dfn.Config):# Additional arguments to base classhost:strport:intclassELKDataset(dfn.DatasetSource):def__init__(self,config,**kwargs):super().__init__(config=config,**kwargs)self.elk_client=elklib(self.config.host,self.config.port)defdataset_name()->str:return'elk-dataset'definfo(self)->dict:return{'description':'My ELK dataset class.','author':'foo@company.org','date':'2021-09-01',}defschema(self)->dict:return{'column_1':'int',...}# this method may also load a row and infer it, like CSVDataset doesdef_generate_examples(self)->Generator:fordocinself.elk_client.scroll_function():# Apply some prep to the documentyielddocAdd new dataset to list of supported datasetsAdd a line toDATASETSindatafun/__init__.py:DATASETS = {
    ...
    "elk-dataset": ELKDataset,
}Use it!ds = dfn.load('elk-dataset', config={
    'host': 'localhost',
    'port': '8888',
})