Package: comparator
COMPARATORComparator is a utility for comparing the results of queries run against
two databases. Future development will include support for APIs, static
files, and more.InstallationpipinstallcomparatorUsageOverviewfromspacklimportdbimportcomparatorascptconf=db.Config()l=db.Postgres(**conf.default)r=db.Postgres(**conf.other_db)query='SELECT * FROM my_table ORDER BY 1'c=cpt.Comparator(l,query,r)c.run_comparisons()[('basic_comp', True)]Included ComparisonsThere are some basic comparisons included, and they can be imported and
passed using constants.fromcomparator.compsimportBASIC_COMP,LEN_COMPc=cpt.Comparator(l,query,r,comps=[BASIC_COMP,LEN_COMP])c.run_comparisons()[('basic_comp', True), ('len_comp', True)]Queries and ExceptionsIt’s possible to run different queries against each database. You can
raise exceptions if that’s your speed.lq='SELECT * FROM my_table ORDER BY 1'rq='SELECT id, uuid, name FROM reporting.my_table ORDER BY 1'comparisons=[BASIC_COMP,LEN_COMP]c=cpt.Comparator(l,lq,r,rq,comps=comparisons)forresultinc.compare():ifnotresult:raiseException('{}check failed!'.format(result.name))Custom ComparisonsYou’ll probably want to define your own comparison checks. You can do so
by defining functions that acceptleftandrightargs, which correspond
to the results of the queries against your “left” and “right” data source,
respectively. Perform whatever magic you like, and return a boolean (or not… your choice).defleft_is_longer(left,right):# Return True if left contains more rows than rightreturnlen(left)>len(right)deftotals_are_equal(left,right):# Return True if sum(left) == sum(right)sl,sr=0,0forrowinleft:sl+=int(row[1])forrowinright:sr+=int(row[1])returnsl==src=cpt.Comparator(l,query,r,comps=[left_is_longer,totals_are_equal])c.run_comparisons()[('left_is_longer', False), ('totals_are_equal', True)]Access Comparator and Query ResultsThe results of both queries and comparisons can be checked using
standard operators, as well as for “truthiness” (ex:failures = [result.name for result in c.compare() if result is False]).Comparisons do not always need to return a boolean. Accessing the
resulting value of such a comparison is simple.deflen_diff(left,right):returnlen(left)-len(right)c=cpt.Comparator(l,query,r,comps=len_diff)res=c.run_comparisons()[0]ifres==0:print('They match')elifres<0:print('Left is shorter by{}'.format(res.result))else:print('Left is longer by{}'.format(res.result))It’s recommended that you use thespacklpackage for instantiating your
“left” and “right” data source objects (pip install spackl). This package
was originally part ofcomparator, and provides the following functionality:Query results are contained in theQueryResultclass, which provides
simple yet powerful ways to look up and access the output of the query.
Data can be retrieved as a dict, list, json string, or pandas DataFrame.
Rows/columns can be accesed by index, attribute, or key. Iterating on
theQueryResultreturns aQueryResultRow, which has the same
lookup functionality, as well as standard operators (<, >, =, etc).fromspacklimportdbconf=db.Config()pg=db.Postgres(**conf.default)res=pg.query(query_string)res# [{'a': 1, 'b': 2, 'c': 3}, {'a': 4, 'b': 5, 'c': 6}, {'a': 7, 'b': 8, 'c': 9}]res.a# (1, 4, 7)res['a']# (1, 4, 7)res[0]# QueryResultRow : (1, 2, 3)res[0].a# 1res[0]['a']# 1res[0][0]# 1res.dict()# {'a': (1, 4, 7), 'b': (2, 5, 8), 'c': (3, 6, 9)}res.list()# [(1, 2, 3), (4, 5, 6), (7, 8, 9)]res.first()# QueryResultRow : (1, 2, 3)These result sets can be used to great effect in comparison callables.
For example, accessing the result of a query as a pandas DataFrame
allows for an endless variety of checks/manipulations do be done on a
single query output.Support is being added tospacklto allow for querying from files and APIs
using the same methods, allowing for easy comparison between many disparate
data sources. Stay tuned.CHANGELOG0.4.0 (2019-03-09)BREAKING - Allsourcemodules and methods have been stripped outFunctionality has been moved to thespacklpackage (pip install spackl)Thecomparatorpackage will expectspacklto be used for allleftandrightdata sources0.4.0rc3 (2018-12-05)Adds better transaction handling in the PostgresDb classCleans up calls to connect() in the Db classes0.4.0rc2 (2018-12-05)BREAKING -QueryPairarguments order has changed (QueryPair(left, lquery, right, rquery))QueryPair,Comparator, andComparatorSetno longer require a “right” Db0.4.0rc1 (2018-11-07)DEPRECATED - thefrom_listmethod onComparatorSetadds theQueryPairclassBREAKING -ComparatorandComparatorSetare instantiated withQueryPairobjectsBREAKING -ComparatorSet.from_dict()requires the dict as the first argumentBREAKING -QueryResult.keys()andQueryResult.values()both return generatorstherquerypassed to aQueryPaircan be formatted with thelqueryquery resultadds theQueryResultColclassadds theappend,pop,extend, andfiltermethods onQueryResultdowngrades pandas version requirement to >=0.22.0improves docstrings onQueryResultmethodsadds slice handling toQueryResultaddsemptyproperty toQueryResult0.3.2 (2018-10-04)adds MANIFEST.in for readme and changes0.3.1 (2018-10-03)addscreds_fileto possible BigQueryDb init kwargs0.3.0 (2018-10-03)DEPRECATED - thequery_dfmethod onBaseDband subclassesDEPRECATED - theoutputkwarg for Comparator resultsadds theexecutemethod onBaseDband subclassesadds theQueryResultandQueryResultRowclassesadds theComparatorSetclassaddslist_tablesanddelete_tablemethods toBigQueryDbcleans up some python 2/3 compatability using six0.2.1 (2018-09-19)officially support Python 2.7, 3.6, and 3.70.2.0 (2018-09-18)addsquery_dfmethods for returning pandas DataFramesaddsoutputkwarg to Comparator to allow calling thequery_dfmethod0.1.0 (2018-09-12)initial release