Package: e6data-python-connector
e6data Python ConnectorIntroductionThe e6data Connector for Python provides an interface for writing Python applications that can connect to e6data and perform operations.DependenciesMake sure to install below dependencies and wheel before install e6data-python-connector.# Amazon Linux / CentOS dependenciesyuminstallpython3-develgcc-c++-y# Ubuntu/Debian dependenciesaptinstallpython3-devg++-y# Pip dependenciespipinstallwheelTo install the Python package, use the command below:pipinstall--no-cache-dire6data-python-connectorPrerequisitesOpen Inbound Port 80 in the Engine Cluster.Limit access to Port 80 according to your organizational security policy. Public access is not encouraged.Access Token generated in the e6data console.Create a ConnectionUse your e6data Email ID as the username and your access token as the password.frome6data_python_connectorimportConnectionusername='<username>'# Your e6data Email ID.password='<password>'# Access Token generated in the e6data console.host='<host>'# IP address or hostname of the cluster to be used.database='<database>'# Database to perform the query on.port=80# Port of the e6data engine.catalog_name='<catalog_name>'conn=Connection(host=host,port=port,username=username,database=database,password=password)Perform a Queries & Get Resultsquery='SELECT * FROM <TABLE_NAME>'# Replace with the query.cursor=conn.cursor(catalog_name=catalog_name)query_id=cursor.execute(query)# The execute function returns a unique query ID, which can be use to abort the query.all_records=cursor.fetchall()forrowinall_records:print(row)To fetch all the records:records=cursor.fetchall()To fetch one record:record=cursor.fetchone()To fetch limited records:limit=500records=cursor.fetchmany(limit)To fetch all the records in buffer to reduce memory consumption:records_iterator=cursor.fetchall_buffer()# Returns generatorforiteminrecords_iterator:print(item)To get the execution plan after query execution:importjsonexplain_response=cursor.explain_analyse()query_planner=json.loads(explain_response.get('planner'))To abort a running query:query_id='<query_id>'# query id from execute function response.cursor.cancel(query_id)Switch database in an existing connection:database='<new_database_name>'# Replace with the new database.cursor=conn.cursor(database,catalog_name)Get Query Time Metricsimportjsonquery='SELECT * FROM <TABLE_NAME>'cursor=conn.cursor(catalog_name)query_id=cursor.execute(query)# execute function returns query id, can be use for aborting the query.all_records=cursor.fetchall()explain_response=cursor.explain_analyse()query_planner=json.loads(explain_response.get('planner'))execution_time=query_planner.get("total_query_time")# In millisecondsqueue_time=query_planner.get("executionQueueingTime")# In millisecondsparsing_time=query_planner.get("parsingTime")# In millisecondsrow_count=query_planner.rowcountGet Schema - a list of Databases, Tables or ColumnsThe following code returns a dictionary of all databases, all tables and all columns connected to the cluster currently in use.
This function can be used without passing database name to get list of all databases.databases=conn.get_schema_names()# To get list of databases.print(databases)database='<database_name>'# Replace with actual database name.tables=conn.get_tables(database=database)# To get list of tables from a database.print(tables)table_name='<table_name>'# Replace with actual table name.columns=conn.get_tables(database=database,table=table_name)# To get the list of columns from a table.columns_with_type=list()"""Getting the column name and type."""forcolumnincolumns:columns_with_type.append(dict(column_name=column.fieldName,column_type=column.fieldType))print(columns_with_type)Code HygieneIt is recommended to clear the cursor, close the cursor and close the connection after running a function as a best practice.
This enhances performance by clearing old data from memory.cursor.clear()# Not needed when aborting a querycursor.close()conn.close()Code ExampleThe following code is an example which combines a few functions described above.frome6data_python_connectorimportConnectionimportjsonusername='<username>'# Your e6data Email ID.password='<password>'# Access Token generated in the e6data console.host='<host>'# IP address or hostname of the cluster to be used.database='<database>'# # Database to perform the query on.port=80# Port of the e6data engine.sql_query='SELECT * FROM <TABLE_NAME>'# Replace with the actual query.catalog_name='<catalog_name>'# Replace with the actual catalog name.conn=Connection(host=host,port=port,username=username,database=database,password=password)cursor=conn.cursor(db_name=database,catalog_name=catalog_name)query_id=cursor.execute(sql_query)all_records=cursor.fetchall()explain_response=cursor.explain_analyse()planner_result=json.loads(explain_response.get('planner'))execution_time=planner_result.get("total_query_time")/1000# Converting into seconds.row_count=cursor.rowcountcolumns=[col[0]forcolincursor.description]# Get the column names and merge them with the results.results=[]forrowinall_records:row=dict(zip(columns,row))results.append(row)print(row)print('Total row count{}, Execution Time (seconds):{}'.format(row_count,execution_time))cursor.clear()cursor.close()conn.close()