Package: lambda-pyathena
PyAthenaPyAthena is a PythonDB API 2.0 (PEP 249)compliant client forAmazon Athena.lambda-pyathenalambda-pyathena is a fork of PyAthena that simply removes boto3 and botocore from the install-requires,
resulting in an AWS Lambda friendly package.RequirementsPythonCPython 2,7, 3.5, 3.6, 3.7Installation$pipinstalllambda-pyathenaExtra packages:PackageInstall commandVersionPandaspip install PyAthena[Pandas]>=0.24.0SQLAlchemypip install PyAthena[SQLAlchemy]>=1.0.0, <2.0.0UsageBasic usagefrompyathenaimportconnectcursor=connect(aws_access_key_id='YOUR_ACCESS_KEY_ID',aws_secret_access_key='YOUR_SECRET_ACCESS_KEY',s3_staging_dir='s3://YOUR_S3_BUCKET/path/to/',region_name='us-west-2').cursor()cursor.execute("SELECT * FROM one_row")print(cursor.description)print(cursor.fetchall())Cursor iterationfrompyathenaimportconnectcursor=connect(aws_access_key_id='YOUR_ACCESS_KEY_ID',aws_secret_access_key='YOUR_SECRET_ACCESS_KEY',s3_staging_dir='s3://YOUR_S3_BUCKET/path/to/',region_name='us-west-2').cursor()cursor.execute("SELECT * FROM many_rows LIMIT 10")forrowincursor:print(row)Query with parameterSupportedDB API paramstyleis onlyPyFormat.PyFormatonly supportsnamed placeholderswith old%operator style and parameters specify dictionary format.frompyathenaimportconnectcursor=connect(aws_access_key_id='YOUR_ACCESS_KEY_ID',aws_secret_access_key='YOUR_SECRET_ACCESS_KEY',s3_staging_dir='s3://YOUR_S3_BUCKET/path/to/',region_name='us-west-2').cursor()cursor.execute("""
               SELECT col_string FROM one_row_complex
               WHERE col_string =%(param)s""",{'param':'a string'})print(cursor.fetchall())if%character is contained in your query, it must be escaped with%%like the following:SELECTcol_stringFROMone_row_complexWHEREcol_string=%(param)sORcol_stringLIKE'a%%'SQLAlchemyInstall SQLAlchemy withpip install"SQLAlchemy>=1.0.0,<2.0.0"orpip install PyAthena[SQLAlchemy].
Supported SQLAlchemy is 1.0.0 or higher and less than 2.0.0.fromurllib.parseimportquote_plus# PY2: from urllib import quote_plusfromsqlalchemy.engineimportcreate_enginefromsqlalchemy.sql.expressionimportselectfromsqlalchemy.sql.functionsimportfuncfromsqlalchemy.sql.schemaimportTable,MetaDataconn_str='awsathena+rest://{aws_access_key_id}:{aws_secret_access_key}@athena.{region_name}.amazonaws.com:443/'\'{schema_name}?s3_staging_dir={s3_staging_dir}'engine=create_engine(conn_str.format(aws_access_key_id=quote_plus('YOUR_ACCESS_KEY_ID'),aws_secret_access_key=quote_plus('YOUR_SECRET_ACCESS_KEY'),region_name='us-west-2',schema_name='default',s3_staging_dir=quote_plus('s3://YOUR_S3_BUCKET/path/to/')))many_rows=Table('many_rows',MetaData(bind=engine),autoload=True)print(select([func.count('*')],from_obj=many_rows).scalar())The connection string has the following format:awsathena+rest://{aws_access_key_id}:{aws_secret_access_key}@athena.{region_name}.amazonaws.com:443/{schema_name}?s3_staging_dir={s3_staging_dir}&...If you do not specifyaws_access_key_idandaws_secret_access_keyusing instance profile or boto3 configuration file:awsathena+rest://:@athena.{region_name}.amazonaws.com:443/{schema_name}?s3_staging_dir={s3_staging_dir}&...NOTE:s3_staging_dirrequires quote. Ifaws_access_key_id,aws_secret_access_keyand other parameter contain special characters, quote is also required.PandasMinimal example for Pandas DataFrame:frompyathenaimportconnectimportpandasaspdconn=connect(aws_access_key_id='YOUR_ACCESS_KEY_ID',aws_secret_access_key='YOUR_SECRET_ACCESS_KEY',s3_staging_dir='s3://YOUR_S3_BUCKET/path/to/',region_name='us-west-2')df=pd.read_sql("SELECT * FROM many_rows",conn)print(df.head())As Pandas DataFrame:frompyathenaimportconnectfrompyathena.utilimportas_pandascursor=connect(aws_access_key_id='YOUR_ACCESS_KEY_ID',aws_secret_access_key='YOUR_SECRET_ACCESS_KEY',s3_staging_dir='s3://YOUR_S3_BUCKET/path/to/',region_name='us-west-2').cursor()cursor.execute("SELECT * FROM many_rows")df=as_pandas(cursor)print(df.describe())If you want to use PandasDataFrame objectdirectly, you can usePandasCursor.AsynchronousCursorAsynchronousCursor is a simple implementation using the concurrent.futures package.
Python 2.7 usesbackport of the concurrent.futurespackage.
This cursor is notDB API 2.0 (PEP 249)compliant.You can use the AsynchronousCursor by specifying thecursor_classwith the connect method or connection object.frompyathenaimportconnectfrompyathena.async_cursorimportAsyncCursorcursor=connect(s3_staging_dir='s3://YOUR_S3_BUCKET/path/to/',region_name='us-west-2',cursor_class=AsyncCursor).cursor()frompyathena.connectionimportConnectionfrompyathena.async_cursorimportAsyncCursorcursor=Connection(s3_staging_dir='s3://YOUR_S3_BUCKET/path/to/',region_name='us-west-2',cursor_class=AsyncCursor).cursor()It can also be used by specifying the cursor class when calling the connection object’s cursor method.frompyathenaimportconnectfrompyathena.async_cursorimportAsyncCursorcursor=connect(s3_staging_dir='s3://YOUR_S3_BUCKET/path/to/',region_name='us-west-2').cursor(AsyncCursor)frompyathena.connectionimportConnectionfrompyathena.async_cursorimportAsyncCursorcursor=Connection(s3_staging_dir='s3://YOUR_S3_BUCKET/path/to/',region_name='us-west-2').cursor(AsyncCursor)The default number of workers is 5 or cpu number * 5.
If you want to change the number of workers you can specify like the following.frompyathenaimportconnectfrompyathena.async_cursorimportAsyncCursorcursor=connect(s3_staging_dir='s3://YOUR_S3_BUCKET/path/to/',region_name='us-west-2',cursor_class=AsyncCursor).cursor(max_workers=10)The execute method of the AsynchronousCursor returns the tuple of the query ID and thefuture object.frompyathenaimportconnectfrompyathena.async_cursorimportAsyncCursorcursor=connect(s3_staging_dir='s3://YOUR_S3_BUCKET/path/to/',region_name='us-west-2',cursor_class=AsyncCursor).cursor()query_id,future=cursor.execute("SELECT * FROM many_rows")The return value of thefuture objectis anAthenaResultSetobject.
This object has an interface that can fetch and iterate query results similar to synchronous cursors.
It also has information on the result of query execution.frompyathenaimportconnectfrompyathena.async_cursorimportAsyncCursorcursor=connect(s3_staging_dir='s3://YOUR_S3_BUCKET/path/to/',region_name='us-west-2',cursor_class=AsyncCursor).cursor()query_id,future=cursor.execute("SELECT * FROM many_rows")result_set=future.result()print(result_set.state)print(result_set.state_change_reason)print(result_set.completion_date_time)print(result_set.submission_date_time)print(result_set.data_scanned_in_bytes)print(result_set.execution_time_in_millis)print(result_set.output_location)print(result_set.description)forrowinresult_set:print(row)frompyathenaimportconnectfrompyathena.async_cursorimportAsyncCursorcursor=connect(s3_staging_dir='s3://YOUR_S3_BUCKET/path/to/',region_name='us-west-2',cursor_class=AsyncCursor).cursor()query_id,future=cursor.execute("SELECT * FROM many_rows")result_set=future.result()print(result_set.fetchall())A query ID is required to cancel a query with the AsynchronousCursor.frompyathenaimportconnectfrompyathena.async_cursorimportAsyncCursorcursor=connect(s3_staging_dir='s3://YOUR_S3_BUCKET/path/to/',region_name='us-west-2',cursor_class=AsyncCursor).cursor()query_id,future=cursor.execute("SELECT * FROM many_rows")cursor.cancel(query_id)NOTE: The cancel method of thefuture objectdoes not cancel the query.PandasCursorPandasCursor directly handles the CSV file of the query execution result output to S3.
This cursor is to download the CSV file after executing the query, and then loaded intoDataFrame object.
Performance is better than fetching data with a cursor.You can use the PandasCursor by specifying thecursor_classwith the connect method or connection object.frompyathenaimportconnectfrompyathena.pandas_cursorimportPandasCursorcursor=connect(s3_staging_dir='s3://YOUR_S3_BUCKET/path/to/',region_name='us-west-2',cursor_class=PandasCursor).cursor()frompyathena.connectionimportConnectionfrompyathena.pandas_cursorimportPandasCursorcursor=Connection(s3_staging_dir='s3://YOUR_S3_BUCKET/path/to/',region_name='us-west-2',cursor_class=PandasCursor).cursor()It can also be used by specifying the cursor class when calling the connection object’s cursor method.frompyathenaimportconnectfrompyathena.pandas_cursorimportPandasCursorcursor=connect(s3_staging_dir='s3://YOUR_S3_BUCKET/path/to/',region_name='us-west-2').cursor(PandasCursor)frompyathena.connectionimportConnectionfrompyathena.pandas_cursorimportPandasCursorcursor=Connection(s3_staging_dir='s3://YOUR_S3_BUCKET/path/to/',region_name='us-west-2').cursor(PandasCursor)The as_pandas method returns aDataFrame object.frompyathenaimportconnectfrompyathena.pandas_cursorimportPandasCursorcursor=connect(s3_staging_dir='s3://YOUR_S3_BUCKET/path/to/',region_name='us-west-2',cursor_class=PandasCursor).cursor()df=cursor.execute("SELECT * FROM many_rows").as_pandas()print(df.describe())print(df.head())Support fetch and iterate query results.frompyathenaimportconnectfrompyathena.pandas_cursorimportPandasCursorcursor=connect(s3_staging_dir='s3://YOUR_S3_BUCKET/path/to/',region_name='us-west-2',cursor_class=PandasCursor).cursor()cursor.execute("SELECT * FROM many_rows")print(cursor.fetchone())print(cursor.fetchmany())print(cursor.fetchall())frompyathenaimportconnectfrompyathena.pandas_cursorimportPandasCursorcursor=connect(s3_staging_dir='s3://YOUR_S3_BUCKET/path/to/',region_name='us-west-2',cursor_class=PandasCursor).cursor()cursor.execute("SELECT * FROM many_rows")forrowincursor:print(row)The DATE and TIMESTAMP of Athena’s data type are returned aspandas.Timestamptype.frompyathenaimportconnectfrompyathena.pandas_cursorimportPandasCursorcursor=connect(s3_staging_dir='s3://YOUR_S3_BUCKET/path/to/',region_name='us-west-2',cursor_class=PandasCursor).cursor()cursor.execute("SELECT col_timestamp FROM one_row_complex")print(type(cursor.fetchone()[0]))# <class 'pandas._libs.tslibs.timestamps.Timestamp'>Execution information of the query can also be retrieved.frompyathenaimportconnectfrompyathena.pandas_cursorimportPandasCursorcursor=connect(s3_staging_dir='s3://YOUR_S3_BUCKET/path/to/',region_name='us-west-2',cursor_class=PandasCursor).cursor()cursor.execute("SELECT * FROM many_rows")print(cursor.state)print(cursor.state_change_reason)print(cursor.completion_date_time)print(cursor.submission_date_time)print(cursor.data_scanned_in_bytes)print(cursor.execution_time_in_millis)print(cursor.output_location)If you want to customize the Dataframe object dtypes and converters, create a converter class like this:frompyathena.converterimportConverterclassCustomPandasTypeConverter(Converter):def__init__(self):super(CustomPandasTypeConverter,self).__init__(mappings=None,types={'boolean':object,'tinyint':float,'smallint':float,'integer':float,'bigint':float,'float':float,'real':float,'double':float,'decimal':float,'char':str,'varchar':str,'array':str,'map':str,'row':str,'varbinary':str,'json':str,})defconvert(self,type_,value):# Not used in PandasCursor.passSpecify the combination of converter functions in the mappings argument and the dtypes combination in the types argument.Then you simply specify an instance of this class in the convertes argument when creating a connection or cursor.frompyathenaimportconnectfrompyathena.pandas_cursorimportPandasCursorcursor=connect(s3_staging_dir='s3://YOUR_S3_BUCKET/path/to/',region_name='us-west-2').cursor(PandasCursor,converter=CustomPandasTypeConverter())frompyathenaimportconnectfrompyathena.pandas_cursorimportPandasCursorcursor=connect(s3_staging_dir='s3://YOUR_S3_BUCKET/path/to/',region_name='us-west-2',converter=CustomPandasTypeConverter()).cursor(PandasCursor)NOTE: PandasCursor handles the CSV file on memory. Pay attention to the memory capacity.AsyncPandasCursorAsyncPandasCursor is an AsyncCursor that can handle Pandas DataFrame.
This cursor directly handles the CSV of query results output to S3 in the same way as PandasCursor.You can use the AsyncPandasCursor by specifying thecursor_classwith the connect method or connection object.frompyathenaimportconnectfrompyathena.async_pandas_cursorimportAsyncPandasCursorcursor=connect(s3_staging_dir='s3://YOUR_S3_BUCKET/path/to/',region_name='us-west-2',cursor_class=AsyncPandasCursor).cursor()frompyathena.connectionimportConnectionfrompyathena.async_pandas_cursorimportAsyncPandasCursorcursor=Connection(s3_staging_dir='s3://YOUR_S3_BUCKET/path/to/',region_name='us-west-2',cursor_class=AsyncPandasCursor).cursor()It can also be used by specifying the cursor class when calling the connection object’s cursor method.frompyathenaimportconnectfrompyathena.async_pandas_cursorimportAsyncPandasCursorcursor=connect(s3_staging_dir='s3://YOUR_S3_BUCKET/path/to/',region_name='us-west-2').cursor(AsyncPandasCursor)frompyathena.connectionimportConnectionfrompyathena.async_pandas_cursorimportAsyncPandasCursorcursor=Connection(s3_staging_dir='s3://YOUR_S3_BUCKET/path/to/',region_name='us-west-2').cursor(AsyncPandasCursor)The default number of workers is 5 or cpu number * 5.
If you want to change the number of workers you can specify like the following.frompyathenaimportconnectfrompyathena.async_pandas_cursorimportAsyncPandasCursorcursor=connect(s3_staging_dir='s3://YOUR_S3_BUCKET/path/to/',region_name='us-west-2',cursor_class=AsyncPandasCursor).cursor(max_workers=10)The execute method of the AsynchronousPandasCursor returns the tuple of the query ID and thefuture object.frompyathenaimportconnectfrompyathena.async_pandas_cursorimportAsyncPandasCursorcursor=connect(s3_staging_dir='s3://YOUR_S3_BUCKET/path/to/',region_name='us-west-2',cursor_class=AsyncPandasCursor).cursor()query_id,future=cursor.execute("SELECT * FROM many_rows")The return value of thefuture objectis anAthenaPandasResultSetobject.
This object has an interface similar toAthenaResultSetObject.frompyathenaimportconnectfrompyathena.async_pandas_cursorimportAsyncPandasCursorcursor=connect(s3_staging_dir='s3://YOUR_S3_BUCKET/path/to/',region_name='us-west-2',cursor_class=AsyncPandasCursor).cursor()query_id,future=cursor.execute("SELECT * FROM many_rows")result_set=future.result()print(result_set.state)print(result_set.state_change_reason)print(result_set.completion_date_time)print(result_set.submission_date_time)print(result_set.data_scanned_in_bytes)print(result_set.execution_time_in_millis)print(result_set.output_location)print(result_set.description)forrowinresult_set:print(row)frompyathenaimportconnectfrompyathena.async_pandas_cursorimportAsyncPandasCursorcursor=connect(s3_staging_dir='s3://YOUR_S3_BUCKET/path/to/',region_name='us-west-2',cursor_class=AsyncPandasCursor).cursor()query_id,future=cursor.execute("SELECT * FROM many_rows")result_set=future.result()print(result_set.fetchall())This object also has an as_pandas method that returns aDataFrame objectsimilar to the PandasCursor.frompyathenaimportconnectfrompyathena.async_pandas_cursorimportAsyncPandasCursorcursor=connect(s3_staging_dir='s3://YOUR_S3_BUCKET/path/to/',region_name='us-west-2',cursor_class=AsyncPandasCursor).cursor()query_id,future=cursor.execute("SELECT * FROM many_rows")result_set=future.result()df=result_set.as_pandas()print(df.describe())print(df.head())The DATE and TIMESTAMP of Athena’s data type are returned aspandas.Timestamptype.frompyathenaimportconnectfrompyathena.async_pandas_cursorimportAsyncPandasCursorcursor=connect(s3_staging_dir='s3://YOUR_S3_BUCKET/path/to/',region_name='us-west-2',cursor_class=AsyncPandasCursor).cursor()query_id,future=cursor.execute("SELECT col_timestamp FROM one_row_complex")result_set=future.result()print(type(result_set.fetchone()[0]))# <class 'pandas._libs.tslibs.timestamps.Timestamp'>As with AsynchronousCursor, you need a query ID to cancel a query.frompyathenaimportconnectfrompyathena.async_pandas_cursorimportAsyncPandasCursorcursor=connect(s3_staging_dir='s3://YOUR_S3_BUCKET/path/to/',region_name='us-west-2',cursor_class=AsyncPandasCursor).cursor()query_id,future=cursor.execute("SELECT * FROM many_rows")cursor.cancel(query_id)Quickly re-run queriesYou can attempt to re-use the results from a previously run query to help save time and money in the cases where your underlying data isn’t changing. Set thecache_sizeparameter ofcursor.execute()to a number larger than 0 to enable cacheing.frompyathenaimportconnectcursor=connect(aws_access_key_id='YOUR_ACCESS_KEY_ID',aws_secret_access_key='YOUR_SECRET_ACCESS_KEY',s3_staging_dir='s3://YOUR_S3_BUCKET/path/to/',region_name='us-west-2').cursor()cursor.execute("SELECT * FROM one_row")# run onceprint(cursor.query_id)cursor.execute("SELECT * FROM one_row",cache_size=10)# re-use earlier resultsprint(cursor.query_id)# You should expect to see the same Query IDResults will only be re-used if the query strings matchexactly, and the query was a DML statement (the assumption being that you always want to re-run queries likeCREATE TABLEandDROP TABLE).The S3 staging directory is not checked, so it’s possible that the location of the results is not in your provideds3_staging_dir.CredentialsSupportBoto3 credentials.Additional environment variable:$exportAWS_ATHENA_S3_STAGING_DIR=s3://YOUR_S3_BUCKET/path/to/$exportAWS_ATHENA_WORK_GROUP=YOUR_WORK_GROUPTestingDepends on the following environment variables:$exportAWS_ACCESS_KEY_ID=YOUR_ACCESS_KEY_ID$exportAWS_SECRET_ACCESS_KEY=YOUR_SECRET_ACCESS_KEY$exportAWS_DEFAULT_REGION=us-west-2$exportAWS_ATHENA_S3_STAGING_DIR=s3://YOUR_S3_BUCKET/path/to/And you need to create a workgroup namedtest-pyathena.Run test$pipinstallpipenv$pipenvinstall--dev$pipenvrunscripts/test_data/upload_test_data.sh$pipenvrunpytest$pipenvrunscripts/test_data/delete_test_data.shRun test multiple Python versions$pipinstallpipenv$pipenvinstall--dev$pipenvrunscripts/test_data/upload_test_data.sh$pyenvlocal3.7.23.6.83.5.72.7.16$pipenvruntox$pipenvrunscripts/test_data/delete_test_data.sh