Package: relativity
multi-index data structuresMotivationWe’re going to take a mental journey of discovery to see why
relativity was written, and how you can use it to simplify
some of the most difficult problems that come up regularly
when programming.  Rather then leaping straight from programming
with python’s standard data structures to programming with
relativistic data structures, we’ll get a running start
by programming in a version of python that is missing
key data structures.  Then, we will draw a line from this
deficient bad version of python to regular python, and
then extend that line on into relativity.Dict to ListImagine programming without hashmaps.  For example, let’s say we have
a list ofRestaurantobjects andCityobjects, and we want to
get how manyRestaurantsare in eachCity.Normally this is simple:restaurants_in_city={}forrestaurantinrestaurants:city=restaurant.cityrestaurants_in_city[city]=restaurants_in_city.get(city,0)+1defget_restaurant_count(city):returnrestaurants_in_city.get(city,0)But, imagine how you would approach the problem if the only available
data structure was a list.cities=[]restaurants_in_city=[]forrestaurantinrestaurants:missing=Trueforidx,cityinenumerate(cities):ifcity==restaurant.city:restaurants_in_city[idx]+=1missing=Falseifmissing:cities.append(restaurant.city)restaurants_in_city.append(1)defget_restaurant_count(city):foridx,city2inenumerate(cities):ifcity==city2:returnrestaurants_in_city[idx]return0Comparing the two examples, there are a few key differences:there are more low value local values (idx)single data structures split into multiple, which must
then be kept in syncthe code is longer, therefore harder to read,
modify, and debugLet’s leave this dystopian data structure wasteland behind
for now and go back to regular python.Dict to M2MThe same differences that showed up when programming with
and without hashmaps will come up again when comparing
programming with single-index hashmaps to relativistic
multi-index hashmaps.Returning to the restaurants and cities example, what if
a restaurant can have multiple locations and we need to
keep track of which cities each restaurant is in,
as well as which restaurants are in each city.Note that we allow a restaurant to have multiple
locations within the same city, so sets must be used
to avoid double counting.restaurants_in_city={}cities_of_restaurant={}forrestaurantinrestaurants:forlocationinrestaurant.locations:restaurants_in_city.setdefault(location.city,set()).add(restaurant)cities_of_restaurant.setdefault(restaurant,set()).add(location.city)defget_restaurants_in_city(city):returnrestaurants_in_city.get(city,set())defget_cities_of_restaurant(restaurant):returncities_of_restaurant.get(restaurant,set())Relativity’s most basic data structure is a many-to-many
mappingM2M.M2Mis a systematic abstraction over
associating every key with a set of values, and every
value with a set of keys.  See howM2Msimplifies
the problem:restaurant_city_m2m=M2M()forrestaurantinrestaurants:forlocationinrestaurant.locations:restaurant_city_m2m.add(restaurant,location.city)get_restaurants_in_city=restaurant_city_m2m.inv.getget_cities_of_restaurant=restaurant_city_m2m.getRecall that the advantages of having single-index hashmaps
were shorter code, with fewer long lived data structures
and fewer local values.M2Mdoesn’t replacedictany more thandictreplaceslist.  Rather it is
a new layer of abstraction that can greatly simplify
a broad class of problems.Is it possible to go further?  Are there higher levels
of abstraction that can represent more complex relationships
in fewer data structures, and be manipulated with fewer
lines of code and intermediate values?M2M to M2MGraphWhere relativity really shines is releiving the programmer
of the burden of keeping data structures consistent with updates.
Let’s consider our restaurant example if we need to be able
to add and remove locations one at a time and still be able
to query.WithM2Mobjects, the problem is doable, but fiddly to
implement:restaurant_location=M2M()location_city=M2M()defadd_location(location):restaurant_location.add(location.restaurant,location)location_city.add(location,location.city)defremove_location(location):dellocation_city[location]delrestaurant_location.inv[location]defrestaurants_in_city(city):restaurants=set()forlocationinlocation_city.inv[city]:forrestaurantinrestaurant_location.inv[location]:restaurants.add(restaurant)returnrestaurantsdefcities_of_restaurant(restaurant):cities=set()forlocationinrestaurant_location[restaurant]:forcityinlocation_city[location]:cities.add(city)returncitiesThis problem can be simplified by stepping up a level of
abstraction.
WhereM2Mis a data structure of keys and values,M2MGraphis a higher-level data structure ofM2Ms.
WithM2MGraph, this problem becomes simple and
intuitive:data=M2MGraph([('restaurant','location'),('location','city')])defadd_location(location):data['restaurant','location','city'].add(location.restaurant,location,location.city)defremove_location(location):data.remove('location',location)defrestaurants_in_city(city):returndata.pairs('city','restaurant').get(city)defcities_of_restaurant(restaurant):returndata.pairs('restaurant','city').get(restaurant)Introducing ChainGraphs are good for representing arbitrary sets of data, but they
are awkward to query overy.M2MChain``ssequences of``M2M``s,where the keys of ``M2Mn are meant to be drawn from the same pool
as the values ofM2Mn - 1.A simple way to construct a chain is with thechainhelper function.students2classes=M2M([('alice','math'),('alice','english'),('bob','english'),('carol','math'),('doug','chemistry')])classmates=chain(students2clases,students2classes.inv)By chaining the student:class map to itself, we can easily
query which students have classes together.>>>classmates.only('alice')M2MChain([M2M([('alice','math'),('alice','english')]),M2M([('math','carol'),('math','alice'),('english','bob'),('english','alice')])])>>>classmates.only('alice').m2ms[1]M2M([('math','carol'),('math','alice'),('english','bob'),('english','alice')])>>>classmates.only('alice').m2ms[1].inv.keys()['bob','carol','alice']Relativity and DataBasesRelativity is excellent at representing many-to-many relationships
from databases which are otherwise awkward to handle.M2M + ORMLet’s consider an example from Django to start.fromdjango.dbimportmodelsclassStudent(models.model):name=models.StringField()classCourse(models.model):name=models.StringField()students=models.ManyToMany(Student)Students take many courses, and each course has many students.Construting anM2Mover these relationships is very natural:fromrelativityimportM2MStudentCourse=Course.students.throughenrollments=M2M(StudentCourse.objects.all().values_list('student','course'))Design PhilosophyDB Feature SetsA typical SQL database, such as PostGres, MySQL, SQLServer, Oracle, or DB2
offers many features which can be split into four categories:relational data model and queriesnetwork protocol and multiple concurrent connectionstransactions, atomic updates, andMVCCpersistent storage, backups, and read replicasLet’s call these “relational”, “network”, “transactional”,
and “persistence” feature sets.“Alternative” DatabasesThe most widely used alternative is probablySQLite.  SQLite
has relational, transactional, and persistence feature sets but does not have
a network protocol.  Instead it must beembeddedas a library inside another application.Another example is the venerableZODB.  ZODB has
network, transactional, and persistence feature sets
but replaces the relational data model
with an object data model.As an extreme example of how less can be more,memcachedhas
only network features.  Data is stored ephemerally in the form of opaque blobs without
any data model.  There is no atomicity of updates: there is no way to ensure that
two writes either both succeed or both fail.The so-called “NoSQL” databases (cassandra,couchdb,mongodb, etc)
generally provide network and persistence features but lack a relational data model
and transactionality.Relativity: Relational à la carteIn this design space, Relativity offers a relational feature set and nothing else.
Relativity allows you to build in-memory data structures that represent relationships
among arbitrary Python objects and then execute queries over those objects and
relationships via a very natural and pythonic API.SQLRelativityresult-setsets and M2Msjoinchain and attachorder bysort and sortedwhere-clauselist comprehensionArchitectureThe fundamental unit of Relativity is therelation, in the form of
theM2M.  All other data structures are
various types ofM2Mcontainers.  AnM2Mis a very simple
data structure that can be represented as two dicts:{key:set(vals)}{val:set(keys)}The main job of theM2Mis to broadcast changes to the
underlyingdictandsetinstances such that they are kept in
sync, and to enumerate all of the key, val pairs.Similarly, the higher order data structures –M2MGraph,M2MChain, andM2MStar– broadcast changes to
underlyingM2Ms and can return and enumerate them.M2MChainandM2MStar: rows of relationsM2MChainandM2MStarare implemented as thin wrappers over alistofM2M.  The main feature they bring provide “row-iteration”.  The difference
between them is how they defined a row.M2MChainrepresents relationships
that connect end-to-end.M2MStarrepresents relationships that all
point to the same base object, similar to astar schema.SharedM2MsAll of the higher order data structures are concerned with the structure
between and amongM2Ms.  The contents within a particularM2Mdoes
not need to maintain any invariants.  That is, if all of theM2Ms within
one of the higher order data structures were scrambled up, the higher order
data structure would still be valid.(Contrast with, if you were to scramble
the key sets and val sets around within anM2M, it would be totally
inconsistent.)This has important consequences, because it means that various instances
ofM2MGraph,M2MChain, andM2MStarmaysharetheir underlyingM2Ms, and continue to update them.  This means that all of these higher
order data structures can be treated as cheap and ephemeral.For example,M2MGraph.chain(*cols)will construct and return a newM2MChainover theM2Ms linking the passed columns.  All that
actually happens here is theM2MGraphis queried for the underlyingM2Ms, then the list ofM2Ms is passed to theM2MChainconstructor which simply holds a reference to them.Another way to think ofM2MGraph,M2MChainandM2MStaris
as cheap views over the underlyingM2Ms.  No matter how much data is in
the underlyingM2Ms, assembling one of these higher order data structures
over top has a fixed, low cost.Relativity & Python EcosystemPandasBoth Relativity and Pandas enable clean extraction of data from a SQL database
to an in-memory data structure which may be further processed.  Both libraries
provide data structures that can easily express queries over the in-memory
data-set that would otherwise be very difficult and tempt a developer to go
back to the database multiple times.This sounds like Relativity and Pandas should be in competition; but, in practice
they are complementary.  Whereas Pandas is excellent at representing tabular
data in rows and columns, Relativity excels at representing the foreign key
relationships that connect rows in different tables.  Pandas makes it easy
to take a SQL result set and further refine it by filtering rows and addding
columns.  Relativity makes it easy to extract the foreign key relationships
among many tables and further refine them by filtering by connectedness and
adding additional relationships.When to UseUsePandasfor doing analysis of data within rows of a table; use
Relativity for doing analysis of the relationships between rows of
different tables.Coming back to the students-and-classes example:classEnrollment(models.Model):student=models.ForeignKey(Student)class=models.ForeignKey(Class)grade=models.FloatField()# 0.0 - 5.0# Pandas is great at determining each students GPAenrollments_data_frame.group_by(['student']).mean()Better TogetherAt a low-level, aPandasSeriesand a RelaitivityM2Mcan
both represent multiple values per key, so it is easy to convert
between the two.>>>importpandas>>>importrelativity>>>s=pandas.Series(data=[1,2,2],index=['a','a','b'])>>>sa1a2b2dtype:int64>>>m2m=relativity.M2M(s.items())>>>m2mM2M([('a',1L),('a',2L),('b',2L)])>>>keys,vals=zip(*m2m.iteritems())>>>s2=pandas.Series(data=vals,index=keys)>>>s2a1a2b2dtype:int64NetworkXNetworkXis the “graph theory library” of Python:“NetworkX is a Python package for the creation, manipulation,
and study of the structure, dynamics, and functions of complex networks.”NetworkXis great at representing arbitrarily connections among a group
of nodes.  Relativity has relationship-centric APIs and data-structures,
wehere theM2Mrepresents a single relationship, andM2MChain,M2MStar, andM2MGraphbuild higher order connections.Underneath, both are backed bydict.