tibbe/ekg — Issue #3: Lightweight client library; monitoring of multiple apps
Author: thoughtpolice
Created: 2012-02-05 17:10:29+00:00
State: open

So, I really like ekg right now, but one thing I've been wondering about for the past few days is if it would be possible to start an EKG server, with arbitrary remote clients being able to connect back to it and display stats.

My main reasoning for this is that 1) it could significantly reduce the dependency chain needed in a project to use ekg (snap pulls in a lot of extra stuff.) It could be reduced significant with a change like this. I always try to be somewhat wary about such huge chains, although most of the time it works fine. There may be a lighter weight HTTP library that would work and alleviate this, but my other reasoning 2) is that eventually it would be nice if you could have an EKG server, with multiple remote clients connected, and the ability to get an overview of all the connected clients - think a Cloud Haskell application, all reporting GC statistics/other stuff to one central monitor server. Even withstanding cloud haskell, this could be very useful if you have multiple Haskell services doing different things - web server, daemon, monitoring applications, etc.

So for example, there could be the regular `ekg` package which supports starting servers for a specific application via `System.Remote.Monitoring` like right now, but the package could also provide, say a simple 'ekg-server' executable you can start. Client applications could then depend on a `ekg-client` library of some sort, with a similar interface like the one provided now, and specify a remote server to report statistics to.

Thinking about it, to avoid the dependency chain, the current code would need to be stratified a bit; for representing say, the core JSON types, and the communication protocol, apart from the networking components, and the server. So there would need to be packages like:

* `ekg-types` or `ekg-common`, which defines the JSON types and protocols used for communication on all fronts.
* `ekg` which depends on `-common`/`-types`, as well as snap, that looks pretty close to how it does now. It offers the server component that you can embed directly into an application right now, and also offers a 'master server' which can handle multiple clients.
* `ekg-client` which depends on `-types`/`-common`, and offers simplistic network functionality that reports GC stats to a remote server.

I realize that having packages like this which must be worked on in lockstep is somewhat troublesome, but it does help in using pieces in isolation.

There are a lot of important other things to take into account for a 'full featured' system, such as aggregate views, and authentication of clients. But I think that just having a server that can handle multiple clients, and splitting off functionality into a client library is a good place to start and not insurmountable by any means. Ad-hoc and one-off applications can always use the simplistic module in the `ekg` package, but having multi-app monitoring could be incredibly useful once you begin having multiple components that operate as separate processes/on different systems.

Johan, are you opposed to these ideas? If not, I may try working on this a bit in the near future and reporting back here. I would like to see `ekg` become a great library for live performance monitoring (an area that is severely lacking in solutions,) but I do understand if the ideas above don't quite fit into your view of what abstractions you want to provide.