Package: mikkoo
APgQtoRabbitMQrelay. Mikkoo is a PgQ consumer that
that publishes to RabbitMQ. In addition, it includes a built in auditing
system that can be used to confirm that all PgQ events are received by
RabbitMQ.Mikkoo is named for the rabbit in the “Clever Rabbit and the Elephant” fable.InstallationMikkoo is available on thePython Package Indexand can be installed viapip:pipinstallmikkooOnce you’ve setupSkytoolsyou may want to
install the optional included utility functions inmikkoo.sqlto make usage
easier.You can do this with a combination ofcurlandpsql:curl-Lhttps://github.com/gmr/mikkoo/blob/main/mikkoo.sql|psqlThis will install multiple stored procedures and an audit table in a mikkoo schema.
Take a look at the DDL to get a good idea of what each funciton is and how it can
be used.PgQ SetupInstallpgqinto your database and create the queue:#CREATEEXTENSIONpgq;CREATEEXTENSION#SELECTpgq.create_queue('test');create_queue--------------1(1row)Ensure thatpgqdis running.PgQ Event to AMQP MappingWhen inserting events into a PgQ queue, thepgq.insert_event/7function
should be used with the following field mappings:PgQ EventAMQPev_typeRouting Keyev_dataMessage bodyev_extra1Exchangeev_extra2Content-Type Propertyev_extra3AMQP Properties[1]ev_extra4Headers[2]ev_timeHeaders.timetampev_txidHeaders.txid[1]AMQP properties should be set as a JSON blob. Values set in theev_extra3field will overwrite the automatically created propertiesapp_id,content_type,correlation_id,headers, andtimestamp.[2]Ifev_extra4is specified and is a JSON key/value dictionary, it will
be assigned to theheadersAMQP property.There is a convenience schema in themikkoo.sqlfile that adds
stored procedures for creating properly formatted mikkoo events in PgQ. In
addition, there is are auditing functions that allow for the creation of an
audit-log of events that were sent to PgQ.AMQP Message PropertiesThe following table defines the available fields that can be set in a JSON blob
in theev_extra3field when inserting an event.PropertyPgSQL Typeapp_idtextcontent_encodingtextcontent_typetextcorrelation_idtextdelivery_modeint2expirationtextmessage_idtextheaderstext/json[3]timestampint4typetextpriorityint4user_idtext[3]headersshould be sent to a key/value JSON blob if specifiedValues assigned in the JSON blob provided toev_extra3take precedence over
the automatically assignedapp_id,content_type,correlation_id,headers, andtimestampvalues created by Mikkoo at processing time.As of 1.0, Mikkoo will automatically add four AMQP headers property values. These
values will not overwrite any values with the same name specified inev_extra4.
overwriting values with the same name, even if they are specified in the
Thesequencevalue is a dynamically generated ID that attempts to provide
fuzzy distributed ordering information. Thetimestampvalue is the ISO-8601
representation of theev_timefield, which is created when an event is added
to PgQ. Thetxidvalue is theev_txidvalue, the PgQ transaction ID for
the event.  These values are added to help provide some level of deterministic
ordering. Theoriginvalue is the hostname of the server that Mikkoo is running
on.Event Insertion ExampleThe following example inserts a JSON blob message body of{"foo": "bar"}that
will be published to thepostgresexchange in RabbitMQ using thetest.routing-keyrouting key. The content type is specified inev_extra2and the AMQPtypemessage property is specified inev_extra3.#SELECTpgq.insert_event('test','test.routing-key','{"foo": "bar"}','postgres','application/json','{"type": "example"}','');insert_event--------------4(1row)When this message is received by RabbitMQ it will have a message body of:{"foo":"bar"}And it will have message properties similar to the following:PropertyExample Valueapp_idmikkoocontent_typeapplication/jsoncorrelation_id0ad6b212-4c84-4eb0-8782-9a44bdfe949fheadersKeyExample Valueoriginmikkoo.domain.comsequence4539586784185129828timestamp2017-02-2306:17:14.471682-00txid41356335556timestamp1449600290typeexampleConfigurationThe Mikkoo configuration file usesYAMLfor markup and allows
for one or more PgQ queue to be processed.If you have a Sentry or a Sentry account, theApplication/sentry_dsnsetting
will turn on sentry exception logging, if theravenclient library is installed.Queues are configured by name under theApplication/workersstanza. The
following example configures two workers for the processing of a queue namedinvoices. Each worker process connects to a local PostgreSQL and RabbitMQ
instance using default credentials.Application:workers:invoices:postgres_url:postgresql://localhost:5432/postgresrabbitmq:host:localhostport:5671vhost:/ssl_options:protocol:2confirm:falseQueue Configuration OptionsThe following table details the configuration options available per queue:KeyDescriptionconfirmEnable/Disable RabbitMQ Publisher Confirmations. Default:Trueconsumer_nameOverwrite the default PgQ consumer name. Default:mikkoomax_failuresMaximum failures before discarding an event. Default:10postgresql_urlThe url for connecting to PostgreSQLrabbitmqData structure for connection parameters to connect to RabbitMQretry_delayHow long in seconds until PgQ emits failed events. Default:10unregisterUnregister a consumer with PgQ on shutdown. Default:Truewait_durationHow long to wait before checking the queue after the last empty
result. Default:10rabbitmqattributes:AttributeDescriptionhostThe hostname or ip address of the RabbitMQ server (str)portThe port of the RabbitMQ server (int)vhostThe virtual host to connect to (str)usernameThe username to connect as (str)passwordThe password to use (str)ssl_optionsOptional: the SSL options for theSSL connection socketheartbeat_intervalOptional: the AMQP heartbeat interval (int) default: 300 secssl_optionsattributes:AttributeDescriptionca_certsThe file path to the concatenated list of CA certificates (str)ca_pathThe directory path to the PEM formatted CA certificates (str)ca_dataThe PEM encoded CA certificates (str)prototcolThe sslPROTOCOL_*enum integer value. Default:2for enumPROTOCOL_TLS(int)certfileThe file path to the PEM formatted certificate file (str)keyfileThe file path to the certificate private key (str)passwordThe password for decrypting thekeyfileprivate key (str)ciphersThe set of available ciphers in the OpenSSL cipher list format (str)Example ConfigurationThe following is an example of a full configuration file:Application:poll_interval:10sentry_dsn:[YOUR SENTRY DSN]statsd:enabled:truehost:localhostport:8125workers:test:confirm:Falseconsumer_name:my_consumermax_failures:5postgres_url:postgresql://localhost:5432/postgresrabbitmq:host:localhostport:5671username:guestpassword:guestssl_options:protocol:2retry_delay:5unregister:Falsewait_duration:5Daemon:user:mikkoopidfile:/var/run/mikkooLogging:version:1formatters:verbose:format:'%(levelname)-10s%(asctime)s%(process)-6d%(processName)-20s%(name)-18s:%(message)s'datefmt:'%Y-%m-%d%H:%M:%S'handlers:console:class:logging.StreamHandlerformatter:verbosedebug_only:Trueloggers:helper:handlers:[console]level:INFOpropagate:truemikkoo:handlers:[console]level:INFOpropagate:truepika:handlers:[console]level:ERRORpropagate:truequeries:handlers:[console]level:ERRORpropagate:truetornado:handlers:[console]level:ERRORpropagate:trueroot:handlers:[console]level:CRITICALpropagate:truedisable_existing_loggers:trueincremental:falseRunning MikkooAfter creating a configuration file for Mikkoo like the one above, simply run the mikkoo application providing the path to the configuration file:mikkoo-cmikkoo.ymlThe application will attempt to daemonize unless you use the-fforeground CLI switch.Mikkoo’s CLI help can be invoked with--helpand yields the following output:$mikkoo-husage:mikkoo[-h][-cCONFIG][-f]MikkooisaPgQtoRabbitMQRelayoptionalarguments:-h,--helpshowthishelpmessageandexit-cCONFIG,--configCONFIGPathtotheconfigurationfile-f,--foregroundRuntheapplicationinteractively