Package: metamorphosis
Metamorphosis: Somewhat opinionated event streamingAuthor: Jefferson Heard, jheard at teamworks dot comCopyright: 2019 Teamworks Inc.License: MIT LicenseMetamorphosis is an event-streaming based microframework that builds on top of Flask, Graphene, Kafka, and Redis. It allows you to define microservices, which consist of a number of mutations (commands), queries, and event consumers.It chooses simplicity of implementation over abstraction, configurability, and flexibility, making the assumptions that you will be using Kafka or Confluent for your streams, Graphene for your GraphQL, and Redis for your result broker and steward of application state. If you think these technologies suck, this is probably not your microframework. If you think this might be your microframework anyway either because it's late and you're tired of looking or because you think it might have potential, please feel free to provide pull requests implementing abstraction layers around Kafka, Redis, or Flask. I'm really happy to take contributions to this work.Also, this framework is based on the famous story about a man who turns into a bug. Plainly I identify deeply with the protagonist, as this is roughly the story of my life. Thus you should assume there are bugs in this code, and you should hunt them down and fix them OR choose to completely ignore them like everyone in Kafka's story. The choice is yours entirely, I can't haveallthe opinions.How does it work?Before you start, you should define your service in terms of whatcommandsandqueriesit accepts, and what events it publishes and subscribes to. At the very least, it publishes an event for every command that it supports, but there may be other events as well. It should also (generally) subscribe at least once to every event generated by a command.  Once you have that structure laid out, see the example and work from that. Or try to fly blind using these instructions that I clearly wrote at 10:39pm in a Holiday Inn while putting off sleep. You can tell which method I think will do you more good. Anyway...Instantiate aMicroserviceobject and give it a service name. This service name also serves as your default topic name on Kafka.Define anEventMutationorAsyncEventMutationfor each command your microservice handles. Event mutations are synchronous and will poll, waiting on the event to show up in Redis with either an Exception or a result.Define themutate_eventclass-method on each mutation, which should authorize and validate the request, then return an instance of the mutation class or some suitable event.Define and registermutation_consumers, which are functions that accept an event and do something with it, such as create an object in a database, create a structured log record, delete spam from your grandmother, etc. Any exception raised by the consumer will be propagated to the caller in the case of a synchronous mutation. Asynchronous mutations will require you to handle exception events in their own way (possibly simply by logging them).Define some queries.Create a flask app.Init the microservice object with it viainit_app.Start the GraphQL server usingflask runorgunicornor however you like to run such thingsUsegregor.py(haha, apropos English-nerd thing, I.R. Smart) to output the default config YAML to a file and modify it and copy it as much as you want to configure your service consumers.Usegregor.pyto start up a microservice consumers as a "node" using the config you just created to control how often the processes kill themselves (very Kafkaesque) and how many processes should be up at a given time.I thought you had opinions?You meanotherthan choosing Kafka, Flask, Graphene, and Redis for you? Why yes, yes I did. Or at least I have a way I envision someone using this thing.Create one GraphQL schema for your entire service-webTechnically this means that your app.py or some such will import ALL of your Microservice objects,init_app()them at once, and then create a Query class that inherits from all the service.Query classes and Mutations class that inherits from all the service.Mutations classes for all services in your application.Allow the GraphQL server to be started in "manager" mode or regular modeIn manager mode, you would inherit from ONE more Query and Mutations class before you create your schema. This would add the metamorphosis.service_web set of mutations to your microservices. They allow you to dynamically adjust the number of consumer processes on each node, on a consumer by consumer basis and see a map of the nodes, microservices, consumers, and processes that are keeping your little ant-farm going (if you use this to make an ant farm, please send me a video of the working farm and a signed letter of indemnity if you happen to commit ant-genocide due to a bug in the framework).The important thing about manager-mode, though, is that it's just another set of GraphQL mutations that produce events on a special "service web topic" on kafka which all consumers on all microservice nodes listen to. This means that you don't want it on the public web.  All of these accept anauthCodeparameter, which is a single code that you can store in an ansible vault, a physical vault, your own eidetic memory, or on a post-it note affixed to your SRE chief's massively over-the-top gaming chair.