Package: dbag
Dbag- Easy time-series metrics and dashboarding===============================================Dbag is a simple Django app to help you remember, graph and create dashboardsfor arbitrary metrics that change over time.It's best used to graph things like:* number of active user accounts* % of users who have logged in today* number of new blog commentsThese are things that are easy to run a query or a quick bit of python todetermine the answer to the question *right now* but that are hard or fuzzy tocalculate for periods in the past. You want to figure out your number every dayand then remember it so you can display fancy graphs and trending (and you wantthese graphs to just work and be pretty). You also want to reduce complexity byusing your existing Django database backend to hold all of the data.Why Dbag?---------Dbag fills a niche not currently well-covered by existing solutions. Dbag issimpler than existing tools because it does less. If you meet the followingconditions, it might be right for you:* You want to collect a small to moderate amount of data and daily resolutionis good enough.* You have many different ways of getting the data, but you want to collect itfrom one place.* You want the simplicity of builtin metric types to get commonly-needed Djangometrics, but you also want the flexibility to define arbitrary pythonfunctions to collect data.* You want simple dashboarding that you can use internally and expose to yourusers without a lot of work (and it should be pretty).* You want to be able to interact with your metrics via Django's ORM if necessary.* You want the option to tie a metric to a specific object in your database. Ifyou have a `Customer` object, you might want the number of active accounts oneach specific customer.What you should use instead---------------------------If that's not what you're looking for, then one of the following is probably abetter option.Operations and SysAdmin Graphing~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~There several great system information graphing applications like, `munin <http://munin-monitoring.org/>`_and `cacti <http://www.cacti.net/>`_ if you want to see CPU usage over time.These are better if you want to know how disk usage is trending across 30different nodes.Capture Events~~~~~~~~~~~~~~`Mixpanel <http://mixpanel.com/>`_, `statsd <https://github.com/etsy/statsd>`_,`Google Analytics <http://www.google.com/analytics/>`_, etc are all better atcapturing events and high-frequency data. Use them if that's all you need. Dbagwould fit in to that equation if you want to regularly slice off or corelatedata from those sources and display the changes over time on a dashboard.Graph Anything~~~~~~~~~~~~~~If you just want to dump any kind of data at any volume in to one system andthen graph it a thousand different ways, you should use`graphite <http://graphite.wikidot.com/>`_. You'll want to put it on a differentserver and you'll need to figure it out, but you'll get as much scalability andflexibility as you need. `dbag` actually works well in combination withgraphite if you'd like to display simple dashboards of summarized date to yourusers.Other Django Dashboards~~~~~~~~~~~~~~~~~~~~~~~There are some other Django dashboarding-ish apps around.* This one isn't meant to be general-purpose, but is used specifically forgrabbing metrics for the `Django projectdashboard <https://github.com/jacobian/django-dev-dashboard>`_. It has veryattractive panels with a optional sparklines and a nice master/defaultlayout. Visual inspiration was taken heavily from this project and `dbag` canbe used to effectively recreate the Django project dashboard..* This sub-project in`djutils <http://charlesleifer.com/docs/djutils/django-utils/dashboard/panels.html>`_effectively re-creates Munin using Django. It allows you to collect andaggregate very granular data using whatever python code you'd like. It doesnot however let you create parameterized panels and metrics (meaning that ifyou wanted to create separate panels for every ``Customer`` in your database,you'd need to write python code registering a panel for each customer.* These exist but `aren't <http://code.google.com/p/django-dashboard/>`_`documented <https://github.com/stefanw/django-dashboard>`_ or`maintained <https://github.com/ojii/django-dashboard>`_.Installation------------1. Get the project source and install it::$ pip install dbag2. Add `dbag` to your tuple of ``INSTALLED_APPS``.3. Add the dbag urls to your ``urls.py``. Eg::urlpatterns = patterns('',url('^dbag/', include('dbag.urls'),)If you're not using `Nexus <https://github.com/dcramer/nexus>`_ then youalso need to add this to ``urls.py`` for automatic ``MetricType``discovery::import dbagdbag.autodiscover()3. Create the database schema::$ ./manage.py syncdbor if you're using `South <http://south.aeracode.org/>`_ ::$ ./manage.py syncdb --migrate4. Configure some initial metrics::$ ./manage.py dbag_init5. If you're already using `Celery <http://celeryproject.org/>`_ thenensure that`celerybeat <http://celery.readthedocs.org/en/latest/userguide/periodic-tasks.html#starting-celerybeat>`_is running. Otherwise, you can run::$ ./manage.py dbag_output_cronjob > /etc/cron.d/dbag_collect_metricsto set up a cron job to collect your metrics every day. You'll need toedit the resulting file to use the correct paths and the correct user.6. If you want to force collection of your first days worth of metrics, you can also run::$ ./manage.py dbag_collect_metricsAlternatively, you can generate some random fake data to demo with using::$ ./manage.py dbag_fake_metrics7. Now start up your devserver, login and visit`http://localhost:8000/dbag/ <http://localhost:8000/dbag/>`_(or wherever you told your ``urls.py`` to point for dbag).Add a New Metric----------------You can add new metrics to start collecting either through the `Nexus<https://github.com/dcramer/nexus>`_ frontend or via the API in python. Eitherway you'll be choosing 5 things to define your metric.**metric_type_label**The label for the type of metric we're collecting. These python subclassesof ``dbag.metric_types.MetricType`` are registered with dbag (with a uniquelabel) and define how a metric is gathered and what options are required togather it. Included examples are an ``ActiveUsersCount`` type that optionallytakes an ORM filter to define a subset of users and a ``MixpanelEvent`` typethat takes an event name and optional properties to slice and records thevalue for the day.**label**The human-readable name of this metric.**slug**A unique slug identifying this metric.**description**An optional long-form description of this metric.**do_collect**Whether or not to collect new values for this metric (default to False).**kwargs**Some MetricTypes take required or optional keyword configuration arguments.In the following example, ``mp_property`` is an optional keyword argument.An example API call to create a metric might be::from dbag import dbag_managerdbag_manager.create_metric('MixpanelEvent',label='superuser comments',slug='superuser_comments',description="number of comments made by superusers",unit_label="comment",unit_label_plural="comments",mp_property="is_superuser=true")Create a New MetricType-----------------------You can add a new MetricType whenever you need to gather/summarize data from anew source. An example would be a MetricType that used github's API to countthe number of open tickets on a specific project. Subclass``dbag.metric_types.MetricType`` with your object, put it in a``dbag_metric_types`` module in one of your ``INSTALLED_APPS`` and then call::from dbag import dbag_managerdbag_manager.register_metric_type(<your label>, <your class>)For now, check the builtin types located at ``dbag.metric_types`` for details.Dbag? Really?-------------A defensible rationalization is that the name is short for "data bag."Is it Awesome?--------------Yes. Increasingly so.TODO- maybe?------------* Add support for Flask and Pyramid (or others?)* Provide a REST API for accessing metrics data