Package: aceql
AceQL HTTPPython Client SDK v5.7.2 - User GuideMarch 2, 2023FundamentalsLicensePython Versions & DB-API 2.0AceQL Server side compatiblityInstallationData transportBest practices for fast response timeDatatypesUsageQuickstartHandling ExceptionsThe error typeMost common AceQL server messagesHTTP Status CodesAdvanced UsageManaging NULL valuesSetting NULL valuesReading NULL valuesTransactionsProxiesTimeoutsBLOB managementBLOB creationBLOB readingManaging BLOB upload progressCalling AceQL Java stored proceduresUsing outer authentication without a password  and with an AceQL Session IDUsing the Metadata Query APIDownloading database schema into a fileAccessing remote database main propertiesGetting Details of Tables and ColumnsFundamentalsThis document describes how to use the AceQL SDK / module and gives some details about how it operates with the AceQL Server side.The AceQL SDK / module allows you to wrap theAceQL HTTP APIsto access remote SQL databases and/or SQL databases in the cloud by simply including standard Python SQL calls in your code, just like you would do for any local database. There is zero learning curve and usage is straightforward.The AceQL Server operation is described inAceQL HTTP Server Installation and Configuration Guide, whose content is sometimes referred to in his User Guide.On the remote side, like the AceQL Server access to the SQL database using Java JDBC, we will sometimes use the JDBC terminology (ResultSet, etc.) in this document. Nevertheless, knowledge of Java or JDBC isnota requirement.LicenseThe SDK is licensed with the liberalApache 2.0license.Python Versions & DB-API 2.0The module supports Python 3.7–3.11.The module provides a SQL interface compliant with the DB-API 2.0 specification described byPEP 249.AceQL Server side compatiblityThis  SDK version requires AceQL HTTP server side v12.2+.Installationpip install aceqlData transportHTTP requests parameters are transported in UTF-8 format and JSON format is used for data and class  transportAll requests are streamed:Output requests (from the client side)  are streamed directly from the socket to the server to avoid buffering any     content bodyInput responses (for the client side)   are streamed directly from the socket to the server to efficiently read     the response bodyLarge content (query results, Blobs/Clobs, etc.) is transferred using files. It is never loaded in memory. Streaming techniques are always used to read and write this content.Best practices for fast response timeEvery HTTP exchange between the client and server side is time-consuming, because the HTTP call is synchronous and waits for the server's responseTry to avoid coding SQL calls inside loops, as this can reduce execution speed. Each SQL call will send an http request and wait for the response from the server.Note that AceQL is optimized as much as possible. A SELECT call returning a huge data volume will not consume memory on the server or client side:  AceQL uses input stream and output stream I/O for data  transfer.Server JDBC ResultSet retrieval is as fast as possible :The ResultSet creation is done once on the server by the executeQuery.The rows are all dumped at once on the servlet output stream by the serverThe client side gets the ResultSet content as a file.All data reading commands are executed locally on the client side with forward-only reading.It is highly recommended to always useCursor.executemanywith autocommit off when you have many rows to INSERT or UPDATE.DatatypesThe main server side JDBC data types for columns are supported:Boolean,Blob/Clob,Integer,Short,Double,Float,BigDecimal,Long,String,Date,Time, andTimestamp.Note that the AceQL module does not allow you to specify data types to use; data types are implicitly chosen by the module.Parameter values are automatically converted to their SQL equivalent. The following Python types can thus be sent to remote server without any problem:Python type / classSQL typeTuple  (None, SqlNullType.<SQL_TYPE>)NULLstrCHAR, VARCHARintINTEGER, or BIGINT,depending on sizeboolBIT, BOOLfloatREALdateDATEdatetimeTIMESTAMPtimeTIMEFile ObjectBLOBNULL and BLOB types are explained in Advanced Usage.This is how SQL types are converted to Python types by default:SQL typePython type / classNULLNoneCHAR, VARCHARstrTINYINT, INTEGERintBIGINTintBOOL, BITboolDATEdateDECIMAL, DOUBLE_PRECISION, FLOAT, NUMERIC, REALfloatTIMESTAMPdatetimeTIMEtimeBLOBResponse streamUsageQuickstartTo use the module, just create aConnectionobject that represents the database:importaceql# URL of the AceQL server, Remote SQL database name & authentication infourl="https://www.acme.com:9443/aceql"database="sampledb"username="user1"password="password1"connection=aceql.connect(url=url,username=username,password=password,database=database)The alternate syntax with parameters in the URL is supported:url="https://www.acme.com:9443/aceql?username=user1&password=password1&database=sampledb"connection=aceql.connect(url=url)The schema of the database is here:sampledbOnce you have aConnection, you can create aCursorobject and call itsexecute()method to perform SQL commands.Following sample shows how to insert 3 new customers using prepared statements:sql="insert into customer values (?, ?, ?, ?, ?, ?, ?, ?)"params=(1,'Sir','John','Smith I','1 Madison Ave','New York','NY 10010','+1 212-586-7001')cursor.execute(sql,params)rows_inserted=cursor.rowcountsql="insert into customer values (?, ?, ?, ?, ?, ?, ?, ?)"params=(2,'Sir','William','Smith II','1 Madison Ave','New York','NY 10010','+1 212-586-7002')cursor.execute(sql,params)rows_inserted+=cursor.rowcountsql="insert into customer values (?, ?, ?, ?, ?, ?, ?, ?)"params=(3,'Sir','William','Smith III','1 Madison Ave','New York','NY 10010','+1 212-586-7003')cursor.execute(sql,params)rows_inserted+=cursor.rowcountprint("rows inserted: "+str(rows_inserted))which returns:rowsinserted:3Thecursor.execute()sends the SQL order and the parameters to the server who executes it on.We view the first inserted customer:sql="select * from customer where customer_id = ?"params=(1,)cursor.execute(sql,params)row=cursor.fetchone()print(row)which returns:(1,'Sir ','John','Smith','1 Madison Ave','New York','NY 10010  ','+1 212-586-7000')The remote result set is downloaded into a file that is read line per line at eachCursor.fetchone()call.We have access to the name and type of each column:fordescincursor.description:print(desc[0]+", "+desc[1])Which returns:customer_id, INTEGER
customer_title, CHAR
fname, VARCHAR
lname, VARCHAR
addressline, VARCHAR
town, VARCHAR
zipcode, CHAR
phone, VARCHARIt's recommended to close theCursorat end of SELECT usage in order to release  the underlying file stream and delete the associated temp file:cursor.close()We view now all the customers and apply good practice to force the closing ofCursor:withclosing(connection.cursor())ascursor:sql="select * from customer where customer_id >= ? order by customer_id"params=(1,)cursor.execute(sql,params)print("rows: "+str(cursor.rowcount))rows=cursor.fetchall()forrowinrows:print(row)Which returns:(1,'Sir ','John','Smith','1 Madison Ave','New York','NY 10010  ','+1 212-586-7001')(2,'Sir ','William','Smith II','1 Madison Ave','New York','NY 10010  ','+1 212-586-7002')(3,'Sir ','William','Smith III','1 Madison Ave','New York','NY 10010  ','+1 212-586-7003')rows:3At end of our session, it is highly recommended to close theConnection:# Make sure connection is always closed in order to close and release# server connection into the pool:connection.close()Handling ExceptionsExcept forTypeError, Exceptions thrown are always an instance ofErrorTheErrorcontains 5 pieces of information:InfoDescriptionReasonThe error message. Retrieved withreasonproperty.Error TypeSee below for description. Retrieved witherror_typeproperty.ExceptionThe originalExceptionthat is the cause, if any. Retrieved withcauseproperty.Http Status CodeSee below for description. Retrieved withhttp_status_codeproperty.Server ExceptionThe Java Exception Stack Trace thrown on server side,  if any. Retrieved withremote_stack_traceproperty.The error typeThe error type allows you to get the type of error, and where the error occurred. It is retrieved with theError.error_typeproperty.Error Type  ValueDescription0The error occurred locally on the client side.   Seehttp_status_codeproperty for more info.  Typical cases: no Internet connection, proxy  authentication required.1The error is due to a JDBC Exception.  It was raised by the remote JDBC Driver and is rerouted  by AceQL as is.  The JDBC error message is accessible viareasonproperty.  Typical case: an error in the SQL statement.Examples: wrong table or column name.2The error was raised by the AceQL Server.  This means that the AceQL Server expected a value or  parameter that was not sent by the client side.  Typical cases: misspelling in URL parameter, missing  required request parameters,  JDBC  Connection expiration, etc.  The detailed error message is accessible viareasonproperty.  See below for most common AceQL Server error  messages.3The AceQL Server forbade the execution of the SQL  statement for a security reason.  For security reasons,reasonproperty gives access to voluntarily vague  details.4The AceQL Server is on failure and raised an  unexpected Java Exception.  The stack track is included and accessible viaremote_stack_traceproperty.Most common AceQL server messagesAceQL Sever  Error Messages   (Error.error_type  = 2)AceQL main  servlet not found in pathAn error occurred  during Blob downloadAn error occurred  during Blob uploadBlob directory  defined in DatabaseConfigurator.getBlobDirectory() does not existConnection is  invalidated (probably expired)Database does not  existInvalid blob_id.  Cannot be used to create a fileInvalid blob_id.  No Blob corresponding to blob_idInvalid  session_idInvalid username  or passwordNo action found  in requestUnable to get a  ConnectionUnknown SQL  action or not supported by softwareHTTP Status CodesThe HTTP Status Code is accessible with theError.http_status_codeproperty. The HTTP Status Code is 200 (OK) on successful completion calls.When an error occurs:If error type is 0, the HTTP Status Code is returned by the client side and may take all possible values in a malformed HTTP call.If error type is > 0, the HTTP Status Code can take one the following values returned by the server side:HTTP  Status CodeDescription400  (BAD REQUEST)Missing element in URL pathMissing request parametersAll JDBC errors raised by the remote JDBC  Driver401  (UNAUTHORIZED)Invalid username or password in connectInvalid session_idThe AceQL Server forbade the execution of  the SQL statement for security reasons404 (NOT_FOUND)BLOB directory does not exist on serverBLOB file not found on server500  (INTERNAL_SERVER_ERROR)The AceQL Server is on failure and raised  an unexpected Java ExceptionAdvanced UsageManaging NULL valuesSetting NULL valuesNULLvalues are handled in a specific way, because the remote server must know the type of theNULLvalue.To create aNULLvalue parameter, create a tuple of 2 elements:First value isNone.Second value is a one of theSqlNullTypeconstants that defines the type of the parameter.This 2 elements tuple is then inserted in the tuple of the prepared statement parameters:sql="insert into customer values (?, ?, ?, ?, ?, ?, ?, ?)"params=(4,'Sir','William','Smith IV','1 Madison Ave','New York','NY 10010',(None,SqlNullType.VARCHAR))cursor.execute(sql,params)Reading NULL valuesANULLcolumn value is returned asNone:sql="select * from customer_3 where customer_id = ? order by customer_id"params=(4,)cursor.execute(sql,params)row=cursor.fetchone()print(row)Execution will return:(4,'Sir ','William','Smith IV','1 Madison Ave','New York','NY 10010  ',None)In this AceQL module version: there is no difference for string columns between a real NULL in the database and the ''NULL" string.TransactionsTransactions are supported by the module. Because the remote server executes JDBC code, client code must follow the JDBC requirement to set the auto commit mode to false prior executing a transaction.This is done withCursor.set_auto_commit(False). It is good practice to always reset auto commit mode to true at end of your transactions. Not that it auto commit mode state is undefined when aConnectionis created withaceql.connect()call.Transaction example:# To do prior transactionself.connection.set_auto_commit(False)cursor=self.connection.cursor()try:# Create a Customersql="insert into customer values (?, ?, ?, ?, ?, ?, ?, ?)"params=(customer_id,'Sir','John','Smith','1 Madison Ave','New York','NY 10010','+1 212-586-7000')cursor.execute(sql,params)# Create an Order for this Customersql="insert into orderlog values ( ?, ?, ?, ?, ?, ?, ?, ?, ? )"the_datetime=datetime.now()the_date=the_datetime.date()# (None, SqlNullType.BLOB) means to set the jpeg_image BLOB# column to NULL on server:params=(customer_id,item_id,"Item Description",9999,the_date,the_datetime,(None,SqlNullType.BLOB),1,2)cursor.execute(sql,params)self.connection.commit()exceptErrorase:print(e)self.connection.rollback()raiseefinally:self.connection.set_auto_commit(True)# Good practicecursor.close()ProxiesThe AceQL module support proxies, using  theproxysyntax ofRequestsThe aceql module uses Requests for HTTP communications with the remote server. All options of a new AceQL connection are passed with theConnectionOptionswrapper.importaceqlfromaceqlimportConnectionOptions# URL of the AceQL server, Remote SQL database name & authentication infourl="https://www.acme.com:9443/aceql"database="sampledb"username="user1"password="password1"proxies={'http':'http://10.10.1.10:3128','https':'http://10.10.1.10:1080',}# We use the ConnectionOptions wrapper class to pass the proxiesconnection_options=ConnectionOptions(proxies=proxies)connection=aceql.connect(url=url,username=username,password=password,database=database,connection_options=connection_options)Authenticated proxies are supported.  Just create anaceql.ProxyAuthinstance and pass it toaceql.connect()with theConnectionOptionswrapper.importaceqlfromaceqlimportConnectionOptionsfromaceqlimportProxyAuthfromsamplesimportmy_proxy# URL of the AceQL server, Remote SQL database name & authentication infourl="https://www.acme.com:9443/aceql"database="sampledb"username="user1"password="password1"proxies={"http":"http://localhost:8081",}# Get the proxy credentials with our own application methodsproxy_username=my_proxy.get_username()proxy_password=my_proxy.get_password()# The AceQL ProxyAuth class allows to define the proxy credentialsauth=ProxyAuth(proxy_username,proxy_password)# We use the ConnectionOptions wrapper class to pass both the proxies & the authconnection_options=ConnectionOptions(proxies=proxies,auth=auth)connection=aceql.connect(url=url,username=username,password=password,database=database,connection_options=connection_options)The AceQL module usesrequests-toolbeltfor authenticated proxy management.TimeoutsUse thetimeoutparameter ofConnectionOptionsto pass a timeout value  in theRequests Timeoutsformat.If no timeout is specified explicitly, requests do not time out.connection_options=ConnectionOptions(timeout=10)connection=aceql.connect(url=url,username=username,password=password,database=database,connection_options=connection_options)BLOB managementThe AceQL module supports BLOB creation and reading. Methods are implemented using streaming techniques to keep low memory consumption. CLOBs are not supported in this version.BLOB creationBLOB creation is supported by passing a tuple with a File Object as parameter of a prepared statement:sql="insert into orderlog values ( ?, ?, ?, ?, ?, ?, ?, ?, ? )"filename=os.getcwd()+sep+"item_1_image.png"fd=open(filename,"rb")# File will be closed by AceQLblob_tuple=(fd,)params=(1,1,"Item 1 Description",9999,datetime.now(),datetime.now().date(),blob_tuple,1,2)cursor.execute(sql,params)BLOB readingBLOB reading is supported throughCursor.get_blob_stream(column_index). The stream can then be read with aforloop that iterates on therespsonse, using syntax provided byRequests:sql="select customer_id, item_id, jpeg_image from orderlog "\"where customer_id = ? and item_id = ?"params=(1,1)cursor.execute(sql,params)row=cursor.fetchone()# You can get BLOB length if you want to use a progress indicatorblob_length=cursor.get_blob_length(2)print("blob length: "+str(blob_length))# Get the stream to the remote BLOBresponse=cursor.get_blob_stream(2)# Download is streamed and writen into filenamefilename=os.path.expanduser("~")+sep+"jpeg_image.jpg"withopen(filename,'wb')asfd:forchunkinresponse.iter_content(chunk_size=2048):fd.write(chunk)stat_info=os.stat(filename)print("file length: "+str(stat_info.st_size))Managing BLOB upload progressYou may want to give your users a progress bar when uploading BLOBs.TheProgressIndicator.Percentproperty allows you to get the current percent of upload. Value will be incremented automatically during upload.To activate the update mechanism:1/ Set the long BLOB length along the File Object in the tuple of the BLOB prepared statement parameter:file_length=os.stat(filename).st_sizefd=open(filename,"rb")blob_tuple=(fd,file_length)2/ Create yourProgressIndicatorinstance and enter it to theConnectioninstance before theCursor.execute(sql,  params)call :progress_indicator=ProgressIndicator()connection.set_progress_indicator(progress_indicator)You then can readProgressIndicator.percentproperty in your watching thread.Code sample:withclosing(connection.cursor())ascursor:filename=os.getcwd()+sep+"item_1_image.jpg"file_length=os.stat(filename).st_sizefd=open(filename,"rb")blob_tuple=(fd,file_length)progress_indicator=ProgressIndicator()connection.set_progress_indicator(progress_indicator)sql="insert into orderlog values ( ?, ?, ?, ?, ?, ?, ?, ?, ? )"params=(1,1,"Item 1 Description",9999,datetime.now(),datetime.now().date(),blob_tuple,1,2)# cursor.execute() uploads BLOB by chunks and increments# ProgressIndicator.percent propertycursor.execute(sql,params)Calling AceQL Java stored proceduresThe AceQL client SDK allows executing a remote server Java class that implements the AceQL Serverorg.kawanfw.sql.api.server.executor.ServerQueryExecutorinterface and that returns the rows of the SELECT.See theorg.kawanfw.sql.api.server.executor.ServerQueryExecutorJavadoc.The usage on the client side is straightforward with theCursor.ExecuteServerQueryAsync()method:server_query_executor_class_name="com.mycompany.MyServerQueryExecutor"my_parameters=[1]cursor.execute_server_query(server_query_executor_class_name,my_parameters)print("cursor.rowcount    : "+str(cursor.rowcount))rows=cursor.fetchall()print("fetchall:")forrowinrows:print(row)cursor.close()Using outer authentication without a password and with an AceQL Session IDSome working environments (Intranet, etc.) require that the client user authenticates himself without a password. Thus, it is not possible for this users to authenticate though the AceQL client SDK.In this case, you may use directly the native HTTPloginAPI to authenticate the users and retrieve thesession_idreturned by the API. Just pass the value of thesession_idto theConnectionOptionsinstance, along with aNonepasswordtoconnect.importaceqlfromaceqlimportConnectionOptionsfromsamplesimportmy_session_id_builder# URL of the AceQL server, Remote SQL database name & authentication info without passwordurl="https://www.acme.com:9443/aceql"database="sampledb"username="user1"# Our application will get the session_id to usesession_id=my_session_id_builder.get_session_id_from_login_api()# We use the ConnectionOptions wrapper to tell to use session_id instead of a passwordconnection_options=ConnectionOptions(session_id=session_id)connection=aceql.connect(url=url,username=username,password=None,database=database,connection_options=connection_options)Using the Metadata Query APIThe metadata API allows:downloading a remote database schema
in HTML or text format,to get a remote database main properties,to get the list of tables,to get the details of each table.It also allows wrapping remote tables, columns, indexes, etc. into
easy to use provided Python classes: Table, Index, Column, etc.First step is to get an instance ofRemoteDatabaseMetaData:remote_database_meta_data=RemoteDatabaseMetaData(connection)Downloading database schema into a fileDownloading a schema into aFileis done through thedb_schema_download(filename)method. See theRemoteDatabaseMetaDatadocumentation:filename=os.path.expanduser("~")+os.sep+"db_schema.html"remote_database_meta_data.db_schema_download(filename)See an example of the built HTML schema:db_schema.out.htmlAccessing remote database main propertiesTheJdbcDatabaseMetaDataclass wraps instance the main value retrieved by a remote server JDBC call tojava.sql.Connection.getMetaData():jdbc_meta_data=remote_database_meta_data.get_jdbc_database_meta_data()print("Major Version: "+str(jdbc_meta_data.getJDBCMajorVersion))print("Minor Version: "+str(jdbc_meta_data.getJDBCMinorVersion))print("IsReadOnly   : "+str(jdbc_meta_data.isReadOnly))Getting Details of Tables and ColumnsSee theRemoteDatabaseMetaDatadocumentation:print("Print the column details of each table:")fortable_nameintable_names:table=remote_database_meta_data.get_table(table_name)print()print("Columns of table: "+table_name)forcolumnintable.columns:print(column)