Package: pandas_drf_tools
Introductionpandas-drf-tools is a set of viewsets, serializers and mixins to allow
usingPandasDataFrames withDjango
REST Frameworksites.InstallationThe package can be installed usingpipfromPyPI:$ pip install pandas-drf-toolsAn you can also install it from source cloning the project’s GitHub
repository:$ git clone git://github.com/abarto/pandas-drf-tools.git
$ cd pandas-drf-tools
$ python setup.py installUsageHow you use pandas-drf-tools depends on the level of integration you
need. The simplest use-case are regular DRF views that expose a
DataFrame. pandas-drf-tools provides several Serializers that turn a
DataFrame into its JSON representation usingto_*methods in the
DataFrame API and a little bit of data processing. You can also parse
(and validate) data sent to the view into a DataFrame using the provided
Serializers. For example:classDataFrameIndexSerializerTestView(views.APIView):defget_serializer_class(self):returnDataFrameIndexSerializerdefget(self,request,*args,**kwargs):sample=get_some_dataframe().sample(20)serializer=self.get_serializer_class()(sample)returnresponse.Response(serializer.data)defpost(self,request,*args,**kwargs):serializer=self.get_serializer_class()(data=request.data)serializer.is_valid(raise_exception=True)data_frame=serializer.validated_datadata={'columns':list(data_frame.columns),'len':len(data_frame)}returnresponse.Response(data)TheAPIViewabove usesDataFrameIndexSerializerto serialize the
DataFrame sample on thegetmethod, and to de-serialize the request
payload on thepostmethod. It also provide basic validation. Here’s
the code forDataFrameIndexSerializer:classDataFrameIndexSerializer(Serializer):defto_internal_value(self,data):try:data_frame=pd.DataFrame.from_dict(data,orient='index').rename(index=int)returndata_frameexceptValueErrorase:raiseValidationError({api_settings.NON_FIELD_ERRORS_KEY:[str(e)]})defto_representation(self,instance):instance=instance.rename(index=str)returninstance.to_dict(orient='index')As you can see, the brunt of the work is done byDataFrame.to_dict.
These are all the Serializers available:DataFrameReadOnlyToDictRecordsSerializer: A read-only (it doesn’t
implementto_internal_value) serializer that usesDataFrame.to_dictwithrecordsorientation.DataFrameListSerializer: A serializer that usesDataFrame.to_dictwithlistorientation for serialization andcolumnsfor
de-serialization.DataFrameIndexSerializer: A serializer that usesDataFrame.to_dictwithindexorientation for serialization
and de-serialization. Due to the restrictions imposed on keys by the
JSON format, the index is converted tostron serialization and
tointon deseralization.DataFrameRecordsSerializer: A serializer that usesDataFrame.to_recordsfor serialization andDataFrame.from_recordsde-serialization.Besides serializers, pandas-drf-tools also provides aGenericDataFrameAPIViewto expose a DataFrame using a view, the same
way DRF’sGenericAPIViewdoes it with Django’s querysets. This class
will rarely be used directly. Same as with DRF, pandas-drf-tools also
provides aGenericDataFrameViewSetclass that, combined with custom
list, retrieve, create, and update mixins turn intoDataFrameViewSet(andReadOnlyDataFrameViewSet) which mimics the behaviour ofModelViewSet.Instead of setting aquerysetfield of overridingget_queryset,
users ofDataFrameViewSetneed to set adataframefield or
override theget_dataframemethod. Another difference is that, by
default, write operations do not change the original dataframe. Thecreate,update, anddestroymethods defined in the mixins
return anewDataFrame based on the one set byget_dataframe. In
order to give the users the chance of doing something with the new
DataFrame, we provide anupdate_dataframecallback that is called
whenever a write operation is called. Take a look at theCreateDataFrameMixinclass:classCreateDataFrameMixin(object):"""
    Adds a row to the dataframe.
    """defcreate(self,request,*args,**kwargs):serializer=self.get_serializer(data=request.data)serializer.is_valid(raise_exception=True)self.perform_create(serializer)headers=self.get_success_headers(serializer.data)returnResponse(serializer.data,status=status.HTTP_201_CREATED,headers=headers)defperform_create(self,serializer):dataframe=self.get_dataframe()returnself.update_dataframe(dataframe.append(serializer.validated_data))defget_success_headers(self,data):try:return{'Location':data[api_settings.URL_FIELD_NAME]}except(TypeError,KeyError):return{}We callappendon the original dataframe and we pass the result ontoupdate_dataframe. The default behaviour ofupdate_dataframeis
just returning whatever was passed onto it, so all operations are
basically read-only. Here’s an example of how to integrate all the
components:importpandasaspdclassTestDataFrameViewSet(DataFrameViewSet):serializer_class=DataFrameRecordsSerializerdefget_dataframe(self):returnpd.read_pickle('test.pkl')defupdate_dataframe(self,dataframe):dataframe.read_pickle('test.pkl')returndataframeThis viewset can then be used the same way as regular DRF viewset. For
instance, we could use a router:fromrest_framework.routersimportDefaultRouterrouter=DefaultRouter()router.register(r'test',TestDataFrameViewSet,base_name='test')The only caveat is that, since there’s no queryset (nor model)
associated with the viewset, DRF cannot guess the base name, so it has
to be set explicitly.That’s everything you need. Now you API is ready to receive regular REST
calls (POST for create, PUT for update, etc.) that will read or change
the DataFrame.Whenever possible, I followed DRF’s existing architecture so most things
should feel natural if you already have experience with the framework.ExampleA complete example that uses the US Census Data is available onGitHub.What’s missing?No unit tests. Although the package is fully functional, I wouldn’t
use it in any production environment yet as I haven’t had time to
fully test it just.No validation. The serializers just use pandas’ methods without
checking payload thoroughly. I’m still looking for ways on improving
this, probably using the columns dtypes to validate each serialized
cell.No filtering backends. If you need filtering, you can override thefilter_dataframemethod, which is does the same as thefilter_querysetmethod. I’m planning on implementing some filters
(like theSearchFilter) to provide guidance if you want to build
your own.No page pagination. OnlyLimitOffsetPaginationis provided.Proper documentation.FeedbackComments, tickets and pull requests are welcomed. You can also reach me
atabarto@machinalis.comif you
have specific questions.