Package: ibmcloudsql
ibmcloudsqlAllows you to run SQL statements in the IBM Cloud on data stored on object storage::Building and testing the library locallySet up Python environmentRunsource ./setup_env.shwhich creates and activates a clean virtual Python environment. It uses Python 2.7 by default. Adapt line 2 inside the script if you want a different version.Install the local code in your Python environmentRun./_install.sh.Test the library locallyCreate a fileibmcloudsql/test_credentials.pywith the following three lines and your according properties:apikey='<your IBM Cloud API key>'
instance_crn='<your Data Engine instance CRN>'
result_location='<COS URI of default result location for your SQL result sets>'
...see details in the template fileRunpython ibmcloudsql/test.py.Packaging and publishing distributionMake sure to increaseversion=...insetup.pybefore creating a new package.Runpackage.sh. It will prompt for user and password that must be authorized for packageibmcloudsqlon pypi.org.Example usageimport ibmcloudsql
my_ibmcloud_apikey = '<your api key here>'
my_instance_crn='<your ibm cloud Data Engine instance CRN here>'
my_target_cos_url='<Cloud Object Storage URL for the SQL result target. Format: cos://<endpoint>/<bucket>/[<prefix>]>'
sqlClient = SQLQuery(my_ibmcloud_apikey, my_instance_crn)
sqlClient.run_sql('SELECT * FROM cos://us-geo/sql/orders.parquet STORED AS PARQUET LIMIT 5 INTO {} STORED AS CSV'.format(my_target_cos_url)).head()Demo notebookYou can use IBM Watson Studio with the followingdemo notebookthat shows some elaborate examples of using various aspects of ibmcloudsql.SQLQuery method listSQLQuery(api_key, instance_crn, target_cos_url=None, token=None, client_info='')Constructorlogon(force=False, token=None)Needs to be called before any other method below. It exchanges theapi_keyset at initialization for a temporary oauth token. The invocation is a No-Op if previous logon is less than 5 minutes ago. You can force logon anyway with optional paramaterforce=True. When you have inititialized the client without anapi_keybut instead specified a customtokenthen you can specify a fresh `token to logon method to update the client with that.submit_sql(sql_text, pagesize=None)ReturnsjobIdas string. Optional pagesize parameter (in rows) for paginated result objects.wait_for_job(jobId)Waits for job to end and returns job completion state (eithercompletedorfailed)get_result(jobId, pagenumber=None)returns SQL result data frame for entire result or for specified page of results.list_results(jobId)Returns a data frame with the list of result objects writtendelete_result(jobId)Deletes all result set objects in cloud object storage for the given jobIdrename_exact_result(jobId)Renames single partitioned query result to exact single object name without folder hierarchy.get_job(jobId)Returns details for the given SQL job as a json objectget_jobs()Returns the list of recent 30 submitted SQL jobs with all details as a data framerun_sql(sql_text)Compound method that callssubmit_sql,wait_for_jobandwait_for_jobin sequenceAsql_ui_link()Returns browser link for Data Engine web console for currently configured instanceget_cos_summary(cos_url)Returns summary for stored number of objects and volume for a given cos url as a jsonlist_cos_objects(cos_url)Returns a data frame with the list of objects found in the given cos urlexport_job_history(cos_url)Exports new jobs as parquet file to the givencos_url.export_tags_for_cos_objects(cos_url, export_target_cos_file)Exports all objects as a parquet file to the givencos_urlthat have tags configured along with the value for each tag.ExceptionsRateLimitedException(message)raised when jobs can't be submitted due to 429 / Plan limit for concurrent queries has been reachedConstructor optionsapi_key: IAM API key. When this parameter is set toNonethen you must specify an own valid IAM otauth token in the parametertoken.instance_crn: Data Engine instance CRN identifiertarget_cos_url: Optional default target URL. Don't use when you want to provide target URL in SQL statement text.token: Optional custom IAM oauth token. When you specify this then you must setapi_keyparameter toNone.client_info: Optional string to identify your client application in IBM Cloud for PD reasons.max_tries: Optional integer to specify maximum attempts when dealing with request rate limit. Default value is1, which means it will through exceptionRateLimitedExceptionwhen response status code is429. It will enableexponential backoffwhen specifying any positive number greater than1. For instance, givenmax_tries=5, assuming it will get response status code429for 4 times until the 5th attempt will get response status code201, the wait time will be2s,4s,8sand16sfor each attempts.LimitationsData Engine Python SDK does not support Pyinstaller.