Package: refractpyhive
PyHivePyHive is a collection of PythonDB-APIandSQLAlchemyinterfaces forPrestoandHive.UsageDB-APIfrompyhiveimportpresto# or import hive or import trinocursor=presto.connect('localhost').cursor()cursor.execute('SELECT * FROM my_awesome_data LIMIT 10')printcursor.fetchone()printcursor.fetchall()DB-API (asynchronous)frompyhiveimporthivefromTCLIService.ttypesimportTOperationStatecursor=hive.connect('localhost').cursor()cursor.execute('SELECT * FROM my_awesome_data LIMIT 10',async=True)status=cursor.poll().operationStatewhilestatusin(TOperationState.INITIALIZED_STATE,TOperationState.RUNNING_STATE):logs=cursor.fetch_logs()formessageinlogs:printmessage# If needed, an asynchronous query can be cancelled at any time with:# cursor.cancel()status=cursor.poll().operationStateprintcursor.fetchall()In Python 3.7asyncbecame a keyword; you can useasync_instead:cursor.execute('SELECT * FROM my_awesome_data LIMIT 10',async_=True)SQLAlchemyFirst install this package to register it with SQLAlchemy (seesetup.py).fromsqlalchemyimport*fromsqlalchemy.engineimportcreate_enginefromsqlalchemy.schemaimport*# Prestoengine=create_engine('presto://localhost:8080/hive/default')# Trinoengine=create_engine('trino://localhost:8080/hive/default')# Hiveengine=create_engine('hive://localhost:10000/default')logs=Table('my_awesome_data',MetaData(bind=engine),autoload=True)printselect([func.count('*')],from_obj=logs).scalar()# Hive + HTTPS + LDAP or basic Authengine=create_engine('hive+https://username:password@localhost:10000/')logs=Table('my_awesome_data',MetaData(bind=engine),autoload=True)printselect([func.count('*')],from_obj=logs).scalar()Note: query generation functionality is not exhaustive or fully tested, but there should be no
problem with raw SQL.Passing session configuration# DB-APIhive.connect('localhost',configuration={'hive.exec.reducers.max':'123'})presto.connect('localhost',session_props={'query_max_run_time':'1234m'})trino.connect('localhost',session_props={'query_max_run_time':'1234m'})# SQLAlchemycreate_engine('presto://user@host:443/hive',connect_args={'protocol':'https','session_props':{'query_max_run_time':'1234m'}})create_engine('trino://user@host:443/hive',connect_args={'protocol':'https','session_props':{'query_max_run_time':'1234m'}})create_engine('hive://user@host:10000/database',connect_args={'configuration':{'hive.exec.reducers.max':'123'}},)# SQLAlchemy with LDAPcreate_engine('hive://user:password@host:10000/database',connect_args={'auth':'LDAP'},)RequirementsInstall usingpip install 'pyhive[hive]'for the Hive interface andpip install 'pyhive[presto]'for the Presto interface.pip install 'pyhive[trino]'for the Trino interfacePyHive works withPython 2.7 / Python 3For Presto: Presto installFor Trino: Trino installFor Hive:HiveServer2daemonChangelogSeehttps://github.com/dropbox/PyHive/releases.ContributingPlease fill out the Dropbox Contributor License Agreement athttps://opensource.dropbox.com/cla/and note this in your pull request.Changes must come with tests, with the exception of trivial things like fixing comments. See .travis.yml for the test environment setup.Notes on project scope:This project is intended to be a minimal Hive/Presto client that does that one thing and nothing else.
Features that can be implemented on top of PyHive, such integration with your favorite data analysis library, are likely out of scope.We prefer having a small number of generic features over a large number of specialized, inflexible features.
For example, the Presto code takes an arbitraryrequests_sessionargument for customizing HTTP calls, as opposed to having a separate parameter/branch for eachrequestsoption.TestingRun the following in an environment with Hive/Presto:./scripts/make_test_tables.sh
virtualenv --no-site-packages env
source env/bin/activate
pip install -e .
pip install -r dev_requirements.txt
py.testWARNING: This drops/creates tables namedone_row,one_row_complex, andmany_rows, plus a
database calledpyhive_test_database.Updating TCLIServiceThe TCLIService module is autogenerated using aTCLIService.thriftfile. To update it, thegenerate.pyfile can be used:python generate.py <TCLIServiceURL>. When left blank, the
version for Hive 2.3 will be downloaded.