Package: ncreplayer
ncreplayerSmall little utility designed to load a CF DSG compliant netCDF file and replay it back onto a Kafka topic in eitherbatchmode orstreammode. Optionally control the timestamps and timedeltas in the original file using configuration parameters.Data is formatted as described in the AVRO schema fileschema.avsc. You can choose to serialize the data asavro,msgpackor the defaultjson.WHY?This utility is designed to make my life easier. It isn't inteded to be used by many people but has been useful for:Playing back data in a netCDF file with different times and intervals than what is defined in the file.Setting up a quick stream of data for load testing.Transitioning parts of large systems that rely on static files to stream processing.Accepting a standardized data format from collaborators (netCDF/CF/DSG) and being able to stream it into larger systems.Data FormatSimplest example{"uid":"1","time":"2019-04-01T00:00:00Z","lat":30.5,"lon":-76.5,}Full Example{"uid":"1","gid":null,"time":"2019-04-01T00:00:00Z","lat":30.5,"lon":-76.5,"z":null,"values":{"salinity":30.2,"temperature":46.5},"meta":""}valuesobjects are optional and are a multi-type AVROmap.metais optional and open-ended. It is intended to carry along metadata to describe thevalues. I recommend usingnco-json. This is useful if the system listening to these messages needs some context about the data, like streaming data to a website. YMMV.ConfigurationThis program usesClickfor the CLI interface.  I probably spent 50% of the time making this utility just playing around with the CLI interface. I have no idea if what I came up with is mind-blowingly awesome or a bunch of crap. It works. Comments are welcome.$ncreplay
Usage:ncreplay[OPTIONS]FILENAMECOMMAND[ARGS]...

Options:--brokersTEXTKafkabrokerstring(commanseparated)[required]--topicTEXTKafkatopictosendthedatato[required]--packing[json|avro|msgpack]Thedatapackingalgorithmtouse--registryTEXTURLtoaSchemaRegistryifavropackingisrequested--uidTEXTVariablename,globalattribute,orvaluetousefortheuidvalues--gidTEXTVariablename,globalattribute,orvaluetouseforthegidvalues--meta/--no-metaIncludethe`nco-json`metadataineachmessage?--helpShowthismessageandexit.

Commands:batchBatchprocessanetCDFfileinchunks,pausingevery[chunk]...streamStreamseachuniquetimestepinthenetCDFfileevery[delta]...batch$ncreplay/path/to/file.ncbatch--help
Usage:ncreplaybatch[OPTIONS]BatchprocessanetCDFfileinchunks,pausingevery[chunk]recordsfor[delta]seconds.Optionallychangethe[starting]timeofthefileand/orchangeeachtimedeltausingthe[factor]and[offset]parameters.

Options:-s,--starting[%Y-%m-%d|%Y-%m-%dT%H:%M:%S|%Y-%m-%d%H:%M:%S]-f,--factorFLOAT-o,--offsetFLOAT-d,--deltaFLOAT-c,--chunkINTEGER--helpShowthismessageandexit.stream$ncreplay/path/to/file.ncstream--help
Usage:ncreplaystream[OPTIONS]StreamseachunqiuetimestepinthenetCDFfileevery[delta]seconds.Optionallyyoucancontrolthe[starting]pointofthefileandthiswillre-calculateallofthetimestampstomatchtheoriginaltimedeltas.

Options:-s,--starting[%Y-%m-%d|%Y-%m-%dT%H:%M:%S|%Y-%m-%d%H:%M:%S]-d,--deltaFLOAT--helpShowthismessageandexitDevelopment / TestingThere are no tests (yet) but you can play around with the options using files included in this repositoryFirst setup a Kafka ecosystem$dockerrun-d--net=host\-eZK_PORT=50000\-eBROKER_PORT=4001\-eREGISTRY_PORT=4002\-eREST_PORT=4003\-eCONNECT_PORT=4004\-eWEB_PORT=4005\-eRUNTESTS=0\-eDISABLE=elastic,hbase\-eDISABLE_JMX=1\-eRUNTESTS=0\-eFORWARDLOGS=0\-eSAMPLEDATA=0\--namencreplayer-testing\landoop/fast-data-dev:1.0.1Then setup a listener$dockerrun-it--rm--net=host\landoop/fast-data-dev:1.0.1\kafka-console-consumer\--bootstrap-serverlocalhost:4001\--topicaxds-ncreplayer-dataNow batch or stream a file:# Batch$ncreplaytests/data/gda_example.ncbatch-d10-c10# Stream$ncreplaytests/data/gda_example.ncstream-d10To test theavropacking, setup a listener that will unpack the data automatically:$dockerrun-it--rm--net=host\landoop/fast-data-dev:1.0.1\kafka-avro-console-consumer\--bootstrap-serverlocalhost:4001\--propertyschema.registry.url=http://localhost:4002\--topicaxds-ncreplayer-dataAnd useavropacking$ncreplay--packingavrotests/data/gda_example.ncbatch-d10-c10