Package: impyla
impylaPython client for HiveServer2 implementations (e.g., Impala, Hive) for
distributed query engines.For higher-level Impala functionality, including a Pandas-like interface over
distributed data sets, see theIbis project.FeaturesHiveServer2 compliant; works with Impala and Hive, including nested dataFullyDB API 2.0 (PEP 249)-compliant Python client (similar to
sqlite or MySQL clients) supporting Python 2.6+ and Python 3.3+.Works with Kerberos, LDAP, SSLSQLAlchemyconnectorConverter topandasDataFrame, allowing easy integration into the
Python data stack (includingscikit-learnandmatplotlib); but see theIbis projectfor a richer
experienceDependenciesRequired:Python 2.7+ or 3.5+six,bitarraythrift==0.16.0thrift_sasl==0.4.3Optional:kerberos>=1.3.0for Kerberos over HTTP support. This also requires Kerberos libraries
to be installed on your system - seeSystem Kerberospandasfor conversion toDataFrameobjects; but see theIbis projectinsteadsqlalchemyfor the SQLAlchemy enginepytestfor running tests;unittest2for testing on Python 2.6System KerberosDifferent systems require different packages to be installed to enable Kerberos support in
Impyla. Some examples of how to install the packages on different distributions follow.Ubuntu:apt-getinstalllibkrb5-devkrb5-userRHEL/CentOS:yuminstallkrb5-libskrb5-develkrb5-serverkrb5-workstationInstallationInstall the latest release withpip:pipinstallimpylaFor the latest (dev) version, install directly from the repo:pipinstallgit+https://github.com/cloudera/impyla.gitor clone the repo:gitclonehttps://github.com/cloudera/impyla.gitcdimpyla
pythonsetup.pyinstallRunning the testsimpyla uses thepytesttoolchain, and depends on the following
environment variables:exportIMPYLA_TEST_HOST=your.impalad.comexportIMPYLA_TEST_PORT=21050exportIMPYLA_TEST_AUTH_MECH=NOSASLTo run the maximal set of tests, runcdpath/to/impyla
py.test--connectimpalaLeave out the--connectoption to skip tests for DB API compliance.UsageImpyla implements thePython DB API v2.0 (PEP 249)database interface
(refer to it for API details):fromimpala.dbapiimportconnectconn=connect(host='my.host.com',port=21050)cursor=conn.cursor()cursor.execute('SELECT * FROM mytable LIMIT 100')printcursor.description# prints the result set's schemaresults=cursor.fetchall()TheCursorobject also exposes the iterator interface, which is buffered
(controlled bycursor.arraysize):cursor.execute('SELECT * FROM mytable LIMIT 100')forrowincursor:print(row)Furthermore theCursorobject returns you information about the columns
returned in the query. This is useful to export your data as a csv file.importcsvcursor.execute('SELECT * FROM mytable LIMIT 100')columns=[datum[0]fordatumincursor.description]targetfile='/tmp/foo.csv'withopen(targetfile,'w',newline='')asoutcsv:writer=csv.writer(outcsv,delimiter=',',quotechar='"',quoting=csv.QUOTE_ALL,lineterminator='\n')writer.writerow(columns)forrowincursor:writer.writerow(row)You can also get back a pandas DataFrame objectfromimpala.utilimportas_pandasdf=as_pandas(cur)# carry df through scikit-learn, for exampleHow do I contribute code?You need to first sign and return anICLAandCCLAbefore we can accept and redistribute your contribution. Once these are submitted you are
free to start contributing to impyla. Submit these toCLA@cloudera.com.FindWe use Github issues to track bugs for this project. Find an issue that you would like to
work on (or file one if you have discovered a new issue!). If no-one is working on it,
assign it to yourself only if you intend to work on it shortly.It's a good idea to discuss your intended approach on the issue. You are much more
likely to have your patch reviewed and committed if you've already got buy-in from the
impyla community before you start.FixNow start coding! As you are writing your patch, please keep the following things in mind:First, please include tests with your patch. If your patch adds a feature or fixes a bug
and does not include tests, it will generally not be accepted. If you are unsure how to
write tests for a particular component, please ask on the issue for guidance.Second, please keep your patch narrowly targeted to the problem described by the issue.
It's better for everyone if we maintain discipline about the scope of each patch. In
general, if you find a bug while working on a specific feature, file a issue for the bug,
check if you can assign it to yourself and fix it independently of the feature. This helps
us to differentiate between bug fixes and features and allows us to build stable
maintenance releases.Finally, please write a good, clear commit message, with a short, descriptive title and
a message that is exactly long enough to explain what the problem was, and how it was
fixed.Please create a pull request on github with your patch.