Package: geolibs-glutemulo
GlutemuloA HA geo socio demo data ingestorUsageRead deexamples files.We use environ vars. SeeEnviron vars file examplefor complete list,
and examples.Using producer to upload data to kafkaSee python examples bellow. You must produce a dict with column_mame:valueUsing the ingestor consumerUseglutodocker and fill enviroment vars.Select the backend usingGLUTEMULO_BACKENDand specific vars for it (database, host, etc).
You can select 2 backends:postgresorcartoSeeEnviron vars file examplefor complete list.Then set:GLUTEMULO_INGESTOR_DATASETTable to upload dataGLUTEMULO_INGESTOR_DATASET_COLUMNSComma separted list of column namesNow, create the table on backend or setGLUTEMULO_INGESTOR_DATASET_DDLandGLUTEMULO_INGESTOR_DATASET_AUTOCREATE=FalseThen configure ingestor for kafka.
First read thepython-kafka docand then use the following vars:GLUTEMULO_INGESTOR_TOPICTopic to useGLUTEMULO_INGESTOR_BOOTSTRAP_SERVERSList of servers to connectGLUTEMULO_INGESTOR_GROUP_IDGroup id.GLUTEMULO_INGESTOR_AUTO_OFFSET_RESETlatest or earliest.GLUTEMULO_INGESTOR_MAX_POLL_RECORDSThe maximum number of records returned in a batch of messagesGLUTEMULO_INGESTOR_FETCH_MIN_BYTESMinimum amount of data the server should return for a fetch request, otherwise wait up to fetch_max_wait_ms for more data to accumulate. Default: 1For the docker, we include aexample docker-compose file.
Remember you can scale with same group_iddocker-composescalegluto=3Run flask demo$FLASK_ENV=developmentflaskrun*Environment:development*Debugmode:on*Runningonhttp://127.0.0.1:5000/(PressCTRL+Ctoquit)*Restartingwithstat*Debuggerisactive!*DebuggerPIN:194-409-049Test$http-jPOSTlocalhost:5000/v1/uno=1dos=2`HTTP/1.0201CREATED
Content-Length:13Content-Type:text/html;charset=utf-8
Date:Thu,02May201914:56:07GMT
Server:Werkzeug/0.15.2Python/3.7.2

DATAReceivedProducer / ConsumerKafka + jsonAsync producer:fromglutemulo.kafka.producerimportJsonKafkaproductor=JsonKafka(bootstrap_servers="localhost:9092")future=productor.produce('simple-topic',dict(dos='BB'))Consumer in batches:fromglutemulo.kafka.consumerimportJsonKafkaconsumer=JsonKafka('simple-topic',bootstrap_servers="localhost:9092")formsginconsumer.consume():formsginmessages:print(msg)Kafka + Avrosync producer:SCHEMA={"type":"record","name":"simpledata","doc":"This is a sample Avro schema to get you started.","fields":[{"name":"name","type":"string"},{"name":"number1","type":"int"},],}SCHEMA_ID=1fromglutemulo.kafka.producerimportAvroKafkaasProducerproductor=Producer(SCHEMA,SCHEMA_ID,bootstrap_servers="localhost:9092")future=productor.produce('simple-topic-avro',dict(name='Un nombre',number1=10))Consumer:fromglutemulo.kafka.consumerimportAvroKafkaasConsumerconsumer=Consumer('simple-topic-avro',SCHEMA,SCHEMA_ID,bootstrap_servers="localhost:9092")formessagesinconsumer.consume():formsginmessages:print(msg)For testingYou can setup a Kafka Consumer using the kafka-console-consumer script that comes with Kafka.$bin/kafka-console-consumer.sh--bootstrap-server192.168.1.240:9092--topicpylog--from-beginning

thisisanawsomelogTesting With KafkaCatYou ca use an application called KafkaCat.After the application is installed we will run it in consumer mode (which is the default).kafkacat-b192.168.240.41:9092-tone-testThis should not show anything yet because we haven't sent anything to our topic yet...To send stuff we can copy any text file into our current directory and send it to our Kafka Topic. In another window, run the following command.$catREADME|kafkacat-b192.168.240.41-tone-testYou should see the output in the first window which has KafkaCat still running in consumer mode.LinksZoonavigator. Use 'zoo1' as connection stringschema-registry-uiRebrowkafka topics uikafka rest proxy