Package: pyathenajdbc
PyAthenaJDBCPyAthenaJDBC is anAmazon Athena JDBC driverwrapper for the PythonDB API 2.0 (PEP 249).RequirementsPythonCPython 3.6, 3.7, 3.8, 3.9JavaJava >= 8 (JDBC 4.2)JDBC driver compatibilityVersionJDBC driver versionVendor< 2.0.0== 1.1.0AWS (Early released JDBC driver. It is incompatible with Simba’s JDBC driver)>= 2.0.0>= 2.0.5SimbaInstallation$pipinstallPyAthenaJDBCExtra packages:PackageInstall commandVersionPandaspip install PyAthenaJDBC[Pandas]>=1.0.0SQLAlchemypip install PyAthenaJDBC[SQLAlchemy]>=1.0.0, <2.0.0UsageBasic usagefrompyathenajdbcimportconnectconn=connect(S3OutputLocation='s3://YOUR_S3_BUCKET/path/to/',AwsRegion='us-west-2')try:withconn.cursor()ascursor:cursor.execute("""
        SELECT * FROM one_row
        """)print(cursor.description)print(cursor.fetchall())finally:conn.close()Cursor iterationfrompyathenajdbcimportconnectconn=connect(S3OutputLocation='s3://YOUR_S3_BUCKET/path/to/',AwsRegion='us-west-2')try:withconn.cursor()ascursor:cursor.execute("""
        SELECT * FROM many_rows LIMIT 10
        """)forrowincursor:print(row)finally:conn.close()Query with parameterSupportedDB API paramstyleis onlyPyFormat.PyFormatonly supportsnamed placeholderswith old%operator style and parameters specify dictionary format.frompyathenajdbcimportconnectconn=connect(S3OutputLocation='s3://YOUR_S3_BUCKET/path/to/',AwsRegion='us-west-2')try:withconn.cursor()ascursor:cursor.execute("""
        SELECT col_string FROM one_row_complex
        WHERE col_string =%(param)s""",{'param':'a string'})print(cursor.fetchall())finally:conn.close()if%character is contained in your query, it must be escaped with%%like the following:SELECTcol_stringFROMone_row_complexWHEREcol_string=%(param)sORcol_stringLIKE'a%%'JVM optionsIn the connect method or connection object, you can specify JVM options with a string array.You can increase the JVM heap size like the following:frompyathenajdbcimportconnectconn=connect(S3OutputLocation='s3://YOUR_S3_BUCKET/path/to/',AwsRegion='us-west-2',jvm_options=['-Xms1024m','-Xmx4096m'])try:withconn.cursor()ascursor:cursor.execute("""
        SELECT * FROM many_rows
        """)print(cursor.fetchall())finally:conn.close()JDBC 4.1If you want to use JDBC 4.1, download the corresponding JDBC driver
and specify the path of the downloaded JDBC driver as the argumentdriver_pathof the connect method or connection object.TheAthenaJDBC41-2.0.7.jaris compatible with JDBC 4.1 and requires JDK 7.0 or later.frompyathenajdbcimportconnectconn=connect(S3OutputLocation='s3://YOUR_S3_BUCKET/path/to/',AwsRegion='us-west-2',driver_path='/path/to/AthenaJDBC41_2.0.7.jar')JDBC driver configuration optionsThe connect method or connection object pass keyword arguments as options to the JDBC driver.
If you want to change the behavior of the JDBC driver,
specify the option as a keyword argument in the connect method or connection object.frompyathenajdbcimportconnectconn=connect(S3OutputLocation='s3://YOUR_S3_BUCKET/path/to/',AwsRegion='us-west-2',LogPath='/path/to/pyathenajdbc/log/',LogLevel='6')For details of the JDBC driver options refer to the official documentation.JDBC Driver Installation and Configuration Guide.NOTE: Option names and values are case-sensitive. The option value is specified as a character string.SQLAlchemyInstall SQLAlchemy withpip installSQLAlchemy>=1.0.0orpip install PyAthenaJDBC[SQLAlchemy].
Supported SQLAlchemy is 1.0.0 or higher and less than 2.0.0.importcontextlibfromurllib.parseimportquote_plusfromsqlalchemy.engineimportcreate_enginefromsqlalchemy.sql.expressionimportselectfromsqlalchemy.sql.functionsimportfuncfromsqlalchemy.sql.schemaimportTable,MetaDataconn_str='awsathena+jdbc://{User}:{Password}@athena.{AwsRegion}.amazonaws.com:443/'\'{Schema}?S3OutputLocation={S3OutputLocation}'engine=create_engine(conn_str.format(User=quote_plus('YOUR_ACCESS_KEY'),Password=quote_plus('YOUR_SECRET_ACCESS_KEY'),AwsRegion='us-west-2',Schema='default',S3OutputLocation=quote_plus('s3://YOUR_S3_BUCKET/path/to/')))try:withcontextlib.closing(engine.connect())asconn:many_rows=Table('many_rows',MetaData(bind=engine),autoload=True)print(select([func.count('*')],from_obj=many_rows).scalar())finally:engine.dispose()The connection string has the following format:awsathena+jdbc://{User}:{Password}@athena.{AwsRegion}.amazonaws.com:443/{Schema}?S3OutputLocation={S3OutputLocation}&driver_path={driver_path}&...If you do not specifyUser(i.e. AWSAccessKeyID) andPassword(i.e. AWSSecretAccessKey) using instance profile credentials or credential profiles file:awsathena+jdbc://:@athena.{Region}.amazonaws.com:443/{Schema}?S3OutputLocation={S3OutputLocation}&driver_path={driver_path}&...NOTE:S3OutputLocationrequires quote. IfUser,Passwordand other parameter contain special characters, quote is also required.PandasAs DataFrameYou can use thepandas.read_sqlto handle the query results as aDataFrame object.frompyathenajdbcimportconnectimportpandasaspdconn=connect(User='YOUR_ACCESS_KEY_ID',Password='YOUR_SECRET_ACCESS_KEY',S3OutputLocation='s3://YOUR_S3_BUCKET/path/to/',AwsRegion='us-west-2',jvm_path='/path/to/jvm')df=pd.read_sql("SELECT * FROM many_rows LIMIT 10",conn)Thepyathena.utilpackage also has helper methods.importcontextlibfrompyathenajdbcimportconnectfrompyathenajdbc.utilimportas_pandaswithcontextlib.closing(connect(S3OutputLocation='s3://YOUR_S3_BUCKET/path/to/'AwsRegion='us-west-2')))asconn:withconn.cursor()ascursor:cursor.execute("""
        SELECT * FROM many_rows
        """)df=as_pandas(cursor)print(df.describe())To SQLYou can usepandas.DataFrame.to_sqlto write records stored in DataFrame to Amazon Athena.pandas.DataFrame.to_sqlusesSQLAlchemy, so you need to install it.importpandasaspdfromurllib.parseimportquote_plusfromsqlalchemyimportcreate_engineconn_str='awsathena+jdbc://:@athena.{AwsRegion}.amazonaws.com:443/'\'{Schema}?S3OutputLocation={S3OutputLocation}&S3Location={S3Location}&compression=snappy'engine=create_engine(conn_str.format(AwsRegion='us-west-2',Schema_name='YOUR_SCHEMA',S3OutputLocation=quote_plus('s3://YOUR_S3_BUCKET/path/to/'),S3Location=quote_plus('s3://YOUR_S3_BUCKET/path/to/')))df=pd.DataFrame({'a':[1,2,3,4,5]})df.to_sql('YOUR_TABLE',engine,schema="YOUR_SCHEMA",index=False,if_exists='replace',method='multi')The location of the Amazon S3 table is specified by theS3Locationparameter in the connection string.
IfS3Locationis not specified,S3OutputLocationparameter will be used. The following rules apply.s3://{S3Location or S3OutputLocation}/{schema}/{table}/The data format only supports Parquet. The compression format is specified by thecompressionparameter in the connection string.CredentialAWS credentials provider chainSeeSupplying and retrieving AWS credentialshttps://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/DefaultAWSCredentialsProviderChain.htmlAWS credentials provider chain that looks for credentials in this order:Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY (RECOMMENDED since they are recognized by all the AWS SDKs and CLI except for .NET), or AWS_ACCESS_KEY and AWS_SECRET_KEY (only recognized by Java SDK)Java System Properties - aws.accessKeyId and aws.secretKeyWeb Identity Token credentials from the environment or containerCredential profiles file at the default location (~/.aws/credentials) shared by all AWS SDKs and the AWS CLICredentials delivered through the Amazon EC2 container service if AWS_CONTAINER_CREDENTIALS_RELATIVE_URI” environment variable is set and security manager has permission to access the variable,Instance profile credentials delivered through the Amazon EC2 metadata serviceIn the connect method or connection object, you can connect by specifying at leastS3OutputLocationandAwsRegion.UserandPasswordare not required if environment variables, credential files, or instance profiles have been set.frompyathenajdbcimportconnectconn=connect(S3OutputLocation='s3://YOUR_S3_BUCKET/path/to/',AwsRegion='us-west-2')TestingDepends on the following environment variables:$exportAWS_ACCESS_KEY_ID=YOUR_ACCESS_KEY_ID$exportAWS_SECRET_ACCESS_KEY=YOUR_SECRET_ACCESS_KEY$exportAWS_DEFAULT_REGION=us-west-2$exportAWS_ATHENA_S3_STAGING_DIR=s3://YOUR_S3_BUCKET/path/to/And you need to create a workgroup namedtest-pyathena-jdbc.Run test$pipinstallpoetry$poetryinstall-v$poetryrunscripts/test_data/upload_test_data.sh$poetryrunpytest$poetryrunscripts/test_data/delete_test_data.shRun test multiple Python versions$pipinstallpoetry$poetryinstall-v$poetryrunscripts/test_data/upload_test_data.sh$pyenvlocal3.9.03.8.63.7.93.6.12$poetryruntox$poetryrunscripts/test_data/delete_test_data.shCode formattingThe code formatting usesblackandisort.Appy format$makefmtCheck format$makechkLicenseThe license of all Python code except JDBC driver isMIT license.JDBC driverFor the license of JDBC driver, please check the following link.JDBC driver release notesJDBC driver licenseJDBC driver noticesJDBC driver third-party licenses