Package: rest-pandas
Django REST PandasDjango REST Framework+pandas= A Model-driven Visualization APIDjango REST Pandas(DRP) provides a simple way to generate and servepandasDataFrames via theDjango REST Framework.  The resulting API can serve up CSV (and a number ofother formats) for consumption by a client-side visualization tool liked3.js.The design philosophy of DRP enforces a strict separation between data and presentation.  This keeps the implementation simple, but also has the nice side effect of making it trivial to provide the source data for your visualizations.  This capability can often be leveraged by sending users to the same URL that your visualization code uses internally to load the data.DRP does not include any JavaScript code, leaving the implementation of interactive visualizations as an exercise for the implementer.  That said, DRP is commonly used in conjunction with thewq.applibrary, which provideswq/chart.jsandwq/pandas.js, a collection of chart functions and data loaders that work well with CSV served by DRP.Live DemoTheclimata-viewerproject uses Django REST Pandas andwq/chart.jsto provide interactive visualizations and spreadsheet downloads.Related WorkThe field of Python-powered data analysis and visualization is growing, and there are a number of similar solutions that may fit your needs better.Django Pandasprovides a custom ORM model manager with pandas support.  By contrast, Django REST Pandas works at theviewlevel, by integrating pandas via custom Django REST Framework serializers and renderers.DRF-CSVprovides straightforward CSV renderers for use with Django REST Framework.  It may be useful if you just want a CSV API and don't have a need for the pandas DataFrame functionality.mpld3provides a direct bridge frommatplotlibtod3.js, complete with seamlessIPythonintegration.  It is restricted to the (large) matplotlib chart vocabularly but should be sufficient for many use cases.Bokehis a complete client-server visualization platform.  It does not leverage d3 or Django, but is notable as a comprehensive, forward-looking approach to addressing similar use cases.The goal of Django REST Pandas is to provide a generic REST API for serving up pandas dataframes.  In this sense, it is similar to the Plot Server in Bokeh, but more generic in that it does not assume any particular visualization format or technology.  Further, DRP is optimized for integration with public-facing Django-powered websites (unlike mpld3 which is primarily intended for use within IPython).In summary, DRP is designed for use cases where:You want to support live spreadsheet downloads as well as interactive visualizations, and/orYou want full control over the client visualization stack in order to integrate it with the rest of your website and/or build process.  This usually means writing JavaScript code by hand.mpld3may be a better choice for data exploration if you are more comfortable with (I)Python and need something that can generate interactive visualizations out of the box.Supported FormatsThe following output formats are provided by default.  These are provided asrenderer classesin order to leverage the content type negotiation built into Django REST Framework.  This means clients can specify a format via:an HTTP "Accept" header (Accept: text/csv),a format parameter (/path?format=csv), ora format extension (/path.csv)The HTTP header and format parameter are enabled by default on every pandas view.  Using the extension requires a custom URL configuration (see below).FormatContent Typepandas DataFrame FunctionNotesHTMLtext/htmlto_html()See notes onHTML outputCSVtext/csvto_csv()TXTtext/plainto_csv()Useful for testing, as most browsers will download a CSV file instead of displaying itJSONapplication/jsonto_json()date_formatandorientcan be provided in URL (e.g./path.json?orient=columns)XLSXapplication/vnd.openxml...sheetto_excel()XLSapplication/vnd.ms-excelto_excel()PNGimage/pngplot()Currently not very customizable, but a simple way to view the data as an image.SVGimage/svgplot()Eventually these could become a fallback for clients that can't handle d3.jsThe underlying implementation is a set ofserializersthat take the normal serializer result and put it into a dataframe.  Then, the includedrenderersgenerate the output using the built in pandas functionality.UsageGetting Started# Recommended: create virtual environment# python3 -m venv venv# . venv/bin/activatepipinstallrest-pandasNOTE:Django REST Pandas relies on pandas, which itself relies on NumPy and other scientific Python libraries written in C.  This is usually fine, since pip can use Python Wheels to install precompiled versions.  If you are having trouble installing DRP due to dependency issues, you may need to pre-install pandas using apt or conda.Usage ExamplesNo ModelThe example below allows you to create a simple API for an existing Pandas DataFrame, e.g. generated from an existing file.# views.pyfromrest_pandasimportPandasSimpleViewimportpandasaspdclassTimeSeriesView(PandasSimpleView):defget_data(self,request,*args,**kwargs):returnpd.read_csv('data.csv')Model-BackedThe example below assumes you already have a Django project set up with a singleTimeSeriesmodel.# views.pyfromrest_pandasimportPandasViewfrom.modelsimportTimeSeriesfrom.serializersimportTimeSeriesSerializer# Short version (leverages default DRP settings):classTimeSeriesView(PandasView):queryset=TimeSeries.objects.all()serializer_class=TimeSeriesSerializer# That's it!  The view will be able to export the model dataset to any of# the included formats listed above.  No further customization is needed to# leverage the defaults.# Long Version and step-by-step explanationclassTimeSeriesView(PandasView):# Assign a default model queryset to the viewqueryset=TimeSeries.objects.all()# Step 1. In response to get(), the underlying Django REST Framework view# will load the queryset and then pass it to the following function.deffilter_queryset(self,qs):# At this point, you can filter queryset based on self.request or other# settings (useful for limiting memory usage).  This function can be# omitted if you are using a filter backend or do not need filtering.returnqs# Step 2. A Django REST Framework serializer class should serialize each# row in the queryset into a simple dict format.  A simple ModelSerializer# should be sufficient for most cases.serializer_class=TimeSeriesSerializer# extends ModelSerializer# Step 3.  The included PandasSerializer will load all of the row dicts# into array and convert the array into a pandas DataFrame.  The DataFrame# is essentially an intermediate format between Step 2 (dict) and Step 4# (output format).  The default DataFrame simply maps each model field to a# column heading, and will be sufficient in many cases.  If you do not need# to transform the dataframe, you can skip to step 4.# If you would like to transform the dataframe (e.g. to pivot or add# columns), you can do so in one of two ways:# A. Create a subclass of PandasSerializer, define a function called# transform_dataframe(self, dataframe) on the subclass, and assign it to# pandas_serializer_class on the view.  You can also use one of the three# provided pivoting serializers (see Advanced Usage below).## class MyCustomPandasSerializer(PandasSerializer):#     def transform_dataframe(self, dataframe):#         dataframe.some_pivot_function(in_place=True)#         return dataframe#pandas_serializer_class=MyCustomPandasSerializer# B. Alternatively, you can create a custom transform_dataframe function# directly on the view.  Again, if no custom transformations are needed,# this function does not need to be defined.deftransform_dataframe(self,dataframe):dataframe.some_pivot_function(in_place=True)returndataframe# NOTE: As the name implies, the primary purpose of transform_dataframe()# is to apply a transformation to an existing dataframe.  In PandasView,# this dataframe is created by serializing data queried from a Django# model.  If you would like to supply your own custom DataFrame from the# start (without using a Django model), you can do so with PandasSimpleView# as shown in the first example.# Step 4. Finally, the provided renderer classes will convert the DataFrame# to any of the supported output formats (see above).  By default, all of# the formats above are enabled.  To restrict output to only the formats# you are interested in, you can define renderer_classes on the view:renderer_classes=[PandasCSVRenderer,PandasExcelRenderer]# You can also set the default renderers for all of your pandas views by# defining the PANDAS_RENDERERS in your settings.py.# Step 5 (Optional).  The default filename may not be particularly useful# for your users.  To override, define get_pandas_filename() on your view.# If a filename is returned, rest_pandas will include the following header:# 'Content-Disposition: attachment; filename="Data Export.xlsx"'defget_pandas_filename(self,request,format):ifformatin('xls','xlsx'):# Use custom filename and Content-Disposition headerreturn"Data Export"# Extension will be appended automaticallyelse:# Default filename from URL (no Content-Disposition header)returnNoneIntegrating with Existing ViewsIf you have an existing viewset, or are otherwise unable to directly subclass one ofPandasSimpleView,PandasView,PandasViewSet, orPandasMixin, you can also integrate the renderers and serializer directly.  The most important thing is to ensure that your serializer class hasMeta.list_serializer_classset toPandasSerializeror a subclass.  Then, make sure that the Pandas renderers are included in yourrenderer options.  See#32and#36for examples.Django Pandas IntegrationYou can also letDjango Pandashandle querying and generating the dataframe, and only use Django REST Pandas for the rendering:# models.pyfromdjango_pandas.managersimportDataFrameManagerclassTimeSeries(models.Model):# ...objects=DataFrameManager()# views.pyfromrest_pandasimportPandasSimpleViewfrom.modelsimportTimeSeriesclassTimeSeriesView(PandasSimpleView):defget_data(self,request,*args,**kwargs):returnTimeSeries.objects.to_timeseries(index='date',)Registering URLs# urls.pyfromdjango.conf.urlsimportpatterns,include,urlfrom.viewsimportTimeSeriesViewurlpatterns=patterns('',url(r'^data',TimeSeriesView.as_view()),)# This is only required to support extension-style formats (e.g. /data.csv)fromrest_framework.urlpatternsimportformat_suffix_patternsurlpatterns=format_suffix_patterns(urlpatterns)The defaultPandasViewwill serve up all of the available data from the provided model in a simple tabular form.  You can also use aPandasViewSetif you are using Django REST Framework'sViewSetsandRouters.Customizing RenderersYou can override the default renderers by settingPANDAS_RENDERERSin yoursettings.py, or by overridingrenderer_classesin your individual view(s).PANDAS_RENDERERSis defined separately from Django REST Framework's ownDEFAULT_RENDERER_CLASSESsetting, in case you want to have DRP-enabled views intermingled with regular DRF views.You can also include DRP renderers inDEFAULT_RENDERER_CLASSES.  In that case, extendPandasMixinor setlist_serializer_classon your serializer.  Otherwise, you may get an error saying the serializer output is not aDataFrame.  In short, there are three paths to getting DRP renderers working with your views:ExtendPandasView,PandasSimpleView, orPandasViewSet, and use thePANDAS_RENDERERSsetting (which defaults to the list above).ExtendPandasMixinand customizeREST_FRAMEWORK['DEFAULT_RENDERER_CLASSES']to add one or morerest_pandasrenderers.Setrenderer_classesexplicitly on the view, and setSerializer.Meta.list_serializer_classtoPandasSerializeror a subclass.  (See#32and#36for examples.)classTimeSeriesView(PandasView):# renderer_classes default to PANDAS_RENDERERS...classTimeSeriesView(PandasMixin,ListAPIView):# renderer_classes default to REST_FRAMEWORK['DEFAULT_RENDERER_CLASSES']...Date FormattingBy default, Django REST Framework will serialize dates as strings before they are processed by the renderer classes.  In many cases, you may want to preserve the dates asdatetimeobjects and let Pandas handle the rendering.  To do this, define an explicitDateTimeFieldorDateFieldon your DRF serializer and setformat=None:# serializers.pyclassTimeSeriesSerializer(serializers.ModelSerializer):date=serializers.DateField(format=None)classMeta:model=TimeSeriesfields='__all__'Alternately, you can disable date serialization globally by settingDATETIME_FORMATand/orDATE_FORMATtoNonein yoursettings.py:# settings.pyDATE_FORMAT=NoneHTML OutputThe HTML renderer provides the ability to create an interactive view that shares the same URL as your data API.  The dataframe is processed byto_html(), then passed toTemplateHTMLRendererwith the following context:context variabledescriptiontableOutput<table>fromto_html()nameView namedescriptionView descriptionurlCurrent URL Path (without parameters)url_paramsURL parametersavailable_formatsArray of allowed extensions (e.g.'csv','json','xlsx')wq_chart_typeRecommended chart type (for use withwq/chartapp.js, see below)As withTemplateHTMLRenderer, the template name is controlled by the view.  If you are using DRP together with thewq framework, you can leverage the defaultmustache/rest_pandas.htmltemplate, which is designed for use with thewq/chartapp.jsplugin.  Otherwise, you will probably want to provide a custom template and/or settemplate_nameon the view.If you need to do a lot of customization, and/or you don't really need the entire dataframe rendered in a<table>, you can always create another view for the interface and make thePandasViewonly handle the API.Note: For backwards compatibility,PandasHTMLRendereris only included in the defaultPANDAS_RENDERERSifrest_pandasis listed in your installed apps.Building Interactive ChartsIn addition to use as a data export tool, DRP is well-suited for creating data API backends for interactive charts.  In particular, DRP can be used withd3.js,wq/pandas.js, andwq/chart.js, to create interactive time series, scatter, and box plot charts - as well as any of the infinite other charting possibilities d3.js provides.To facilitate data API building, the CSV renderer is the default in Django REST Pandas.  While the pandas JSON serializer is improving, the primary reason for making CSV the default is the compactness it provides over JSON when serializing time series data.  The default CSV output from DRP will have single row of column headers, making it suitable as-is for use with e.g.d3.csv().   However, DRP is often used with the custom serializers below to produce a dataframe with nested multi-row column headers.  This is harder to parse withd3.csv()but can be easily processed bywq/pandas.js, an extension to d3.js.// mychart.jsdefine(['d3','wq/pandas','wq/chart'],function(d3,pandas,chart){// Unpivoted data (single-row header)d3.csv("/data.csv",render);// Pivoted data (multi-row header)pandas.get('/data.csv',render);functionrender(error,data){d3.select('svg').selectAll('rect').data(data)// ...}});DRP includes three custom serializers withtransform_dataframe()functions that address common use cases.  These serializer classes can be leveraged by assigning them topandas_serializer_classon your view.  If you are using thewq framework, these serializers can automatically leverage DRP's defaultHTML templatetogether withwq/chartapp.jsto provide interactive charts.  If you are not using the full wq framework, you can still usewq/pandas.jsandwq/chart.jsdirectly with the CSV output of these serializers.For documentation purposes, the examples below assume the following dataset:LocationMeasurementDateValuesite1temperature2016-01-013site1humidity2016-01-0130site2temperature2016-01-014site2temperature2016-01-025PandasUnstackedSerializerPandasUnstackedSerializerunstacksthe dataframe so a few key attributes are listed in a multi-row column header.  This makes it easier to include metadata about e.g. a time series without repeating the same values on every data row.To specify which attributes to use in column headers, define the attributepandas_unstacked_headeron yourModelSerializersubclass.  You will generally also want to definepandas_index, which is a list of metadata fields unique to each row (e.g. the timestamp).# serializers.pyfromrest_frameworkimportserializersfrom.modelsimportTimeSeriesclassTimeSeriesSerializer(serializers.ModelSerializer):classMeta:model=MultiTimeSeriesfields=['date','location','measurement','value']pandas_index=['date']pandas_unstacked_header=['location','measurement']# views.pyfromrest_pandasimportPandasView,PandasUnstackedSerializerfrom.modelsimportTimeSeriesfrom.serializersimportTimeSeriesSerializerclassTimeSeriesView(PandasView):queryset=TimeSeries.objects.all()serializer_class=TimeSeriesSerializerpandas_serializer_class=PandasUnstackedSerializerWith the above example data, this configuration would output a CSV file with the following layout:ValueValueValueLocationsite1site1site2MeasurementtemperaturehumiditytemperatureDate2016-01-0133042016-01-025This could then be processed bywq/pandas.jsinto the following structure:[{"location":"site1","measurement":"temperature","data":[{"date":"2016-01-01","value":3}]},{"location":"site1","measurement":"humidity","data":[{"date":"2016-01-01","value":30}]},{"location":"site2","measurement":"temperature","data":[{"date":"2016-01-01","value":4},{"date":"2016-01-02","value":5}]}]The output ofPandasUnstackedSerializercan be used with thetimeSeries()chart provided bywq/chart.js:define(['d3','wq/pandas','wq/chart'],function(d3,pandas,chart){varsvg=d3.select('svg');varplot=chart.timeSeries();pandas.get('/data/timeseries.csv',function(data){svg.datum(data).call(plot);});});PandasScatterSerializerPandasScatterSerializerunstacks the dataframe and also combines selected attributes to make it easier to plot two measurements against each other in an x-y scatterplot.To specify which attributes to use for the coordinate names, define the attributepandas_scatter_coordon yourModelSerializersubclass.  You can also specify additional metadata attributes to include in the header withpandas_scatter_header.  You will generally also want to definepandas_index, which is a list of metadata fields unique to each row (e.g. the timestamp).# serializers.pyfromrest_frameworkimportserializersfrom.modelsimportTimeSeriesclassTimeSeriesSerializer(serializers.ModelSerializer):classMeta:model=MultiTimeSeriesfields=['date','location','measurement','value']pandas_index=['date']pandas_scatter_coord=['measurement']pandas_scatter_header=['location']# views.pyfromrest_pandasimportPandasView,PandasScatterSerializerfrom.modelsimportTimeSeriesfrom.serializersimportTimeSeriesSerializerclassTimeSeriesView(PandasView):queryset=TimeSeries.objects.all()serializer_class=TimeSeriesSerializerpandas_serializer_class=PandasScatterSerializerWith the above example data, this configuration would output a CSV file with the following layout:temperature-valuehumidity-valuetemperature-valueLocationsite1site1site2Date2014-01-0133042014-01-025This could then be processed bywq/pandas.jsinto the following structure:[{"location":"site1","data":[{"date":"2016-01-01","temperature-value":3,"humidity-value":30}]},{"location":"site2","data":[{"date":"2016-01-01","temperature-value":4},{"date":"2016-01-02","temperature-value":5}]}]The output ofPandasScatterSerializercan be used with thescatter()chart provided bywq/chart.js:define(['d3','wq/pandas','wq/chart'],function(d3,pandas,chart){varsvg=d3.select('svg');varplot=chart.scatter().xvalue(function(d){returnd['temperature-value'];}).yvalue(function(d){returnd['humidity-value'];});pandas.get('/data/scatter.csv',function(data){svg.datum(data).call(plot);});});PandasBoxplotSerializerPandasBoxplotSerializercomputes boxplot statistics (via matplotlib'sboxplot_stats) and pushes the results out via an unstacked dataframe.  The statistics can be aggregated for a specified group column as well as by date.To specify which attribute to use for the group column, define the attributepandas_boxplot_groupon yourModelSerializersubclass.  To specify an attribute to use for date-based grouping, definepandas_boxplot_date.   You will generally also want to definepandas_boxplot_header, which will unstack any metadata columns and exclude them from statistics.# serializers.pyfromrest_frameworkimportserializersfrom.modelsimportTimeSeriesclassTimeSeriesSerializer(serializers.ModelSerializer):classMeta:model=MultiTimeSeriesfields=['date','location','measurement','value']pandas_boxplot_group='site'pandas_boxplot_date='date'pandas_boxplot_header=['measurement']# views.pyfromrest_pandasimportPandasView,PandasBoxplotSerializerfrom.modelsimportTimeSeriesfrom.serializersimportTimeSeriesSerializerclassTimeSeriesView(PandasView):queryset=TimeSeries.objects.all()serializer_class=TimeSeriesSerializerpandas_serializer_class=PandasBoxplotSerializerWith the above example data, this configuration will output a CSV file with the same general structure asPandasUnstackedSerializer, but with thevaluespread across multiple boxplot statistics columns (value-mean,value-q1,value-whishi, etc.).  An optionalgroup` parameter can be added to the query string to switch between various groupings:namepurpose?group=seriesGroup by series (pandas_boxplot_group)?group=series-yearGroup by series, then by year?group=series-monthGroup by series, then by month?group=yearSummarize all data by year?group=monthSummarize all data by monthThe output ofPandasBoxplotSerializercan be used with theboxplot()chart provided bywq/chart.js:define(['d3','wq/pandas','wq/chart'],function(d3,pandas,chart){varsvg=d3.select('svg');varplot=chart.boxplot();pandas.get('/data/boxplot.csv?group=year',function(data){svg.datum(data).call(plot);});});