Package: td-client
Treasure Data API library for PythonRequirementstd-clientsupports the following versions of Python.Python 3.5+PyPyInstallYou can install the releases fromPyPI.$pipinstalltd-clientIt’d be better to installcertifito enable SSL certificate verification.$pipinstallcertifiExamplesPlease see also the examples atTreasure Data Documentation.The td-client documentation is hosted athttps://tdclient.readthedocs.io/,
or you can go directly to theAPI documentation.For information on the parameters that may be used when reading particular
types of data, seeFile import parameters.Listing jobsTreasure Data API key will be read from environment variableTD_API_KEY, if none is given viaapikey=argument passed totdclient.Client.Treasure Data API endpointhttps://api.treasuredata.comis used by default. You can override this with environment variableTD_API_SERVER, which in turn can be overridden viaendpoint=argument passed totdclient.Client. List of available Treasure Data sites and corresponding API endpoints can be foundhere.importtdclientwithtdclient.Client()astd:forjobintd.jobs():print(job.job_id)Running jobsRunning jobs on Treasure Data.importtdclientwithtdclient.Client()astd:job=td.query("sample_datasets","SELECT COUNT(1) FROM www_access",type="hive")job.wait()forrowinjob.result():print(repr(row))Running jobs via DBAPI2td-client-python implementsPEP 0249Python Database API v2.0.
You can use td-client-python with external libraries which supports Database API such likepandas.importpandasimporttdclientdefon_waiting(cursor):print(cursor.job_status())withtdclient.connect(db="sample_datasets",type="presto",wait_callback=on_waiting)astd:data=pandas.read_sql("SELECT symbol, COUNT(1) AS c FROM nasdaq GROUP BY symbol",td)print(repr(data))We offer another package for pandas namedpytdwith some advanced features.
You may prefer it if you need to do complicated things, such like exporting result data to Treasure Data, printing job’s
progress during long execution, etc.Importing dataImporting data into Treasure Data in streaming manner, as similar asfluentdis doing.importsysimporttdclientwithtdclient.Client()astd:forfile_nameinsys.argv[:1]:td.import_file("mydb","mytbl","csv",file_name)WarningImporting data in streaming manner requires certain amount of time to be ready to query since schema update will be
executed with delay.Bulk importImporting data into Treasure Data in batch manner.importsysimporttdclientimportuuidimportwarningsiflen(sys.argv)<=1:sys.exit(0)withtdclient.Client()astd:session_name="session-{}".format(uuid.uuid1())bulk_import=td.create_bulk_import(session_name,"mydb","mytbl")try:forfile_nameinsys.argv[1:]:part_name="part-{}".format(file_name)bulk_import.upload_file(part_name,"json",file_name)bulk_import.freeze()except:bulk_import.delete()raisebulk_import.perform(wait=True)if0<bulk_import.error_records:warnings.warn("detected{}error records.".format(bulk_import.error_records))if0<bulk_import.valid_records:print("imported{}records.".format(bulk_import.valid_records))else:raise(RuntimeError("no records have been imported:{}".format(bulk_import.name)))bulk_import.commit(wait=True)bulk_import.delete()If you want to import data asmsgpackformat, you can write as follows:importioimporttimeimportuuidimportwarningsimporttdclientt1=int(time.time())l1=[{"a":1,"b":2,"time":t1},{"a":3,"b":9,"time":t1}]withtdclient.Client()astd:session_name="session-{}".format(uuid.uuid1())bulk_import=td.create_bulk_import(session_name,"mydb","mytbl")try:_bytes=tdclient.util.create_msgpack(l1)bulk_import.upload_file("part","msgpack",io.BytesIO(_bytes))bulk_import.freeze()except:bulk_import.delete()raisebulk_import.perform(wait=True)# same as the above exampleChanging how CSV and TSV columns are readThetd-clientpackage will generally make sensible choices on how to read
the columns in CSV and TSV data, but sometimes the user needs to override the
default mechanism. This can be done using the optionalfile import
parametersdtypesandconverters.For instance, consider CSV data that starts with the following records:time,col1,col2,col3
1575454204,a,0001,a;b;c
1575454204,b,0002,d;e;fIf that data is read using the defaults, it will produce values that look
like:1575454204,"a",1,"a;b;c"1575454204,"b",2,"d;e;f"that is, an integer, a string, an integer and another string.If the user wants to keep the leading zeroes incol2, then they can
specify the column datatype as string. For instance, usingbulk_import.upload_fileto read data frominput_data:bulk_import.upload_file("part","msgpack",input_data,dtypes={"col2":"str"},)which would produce:1575454204,"a","0001","a;b;c"1575454204,"b","0002","d;e;f"If they also wanted to treatcol3as a sequence of strings, separated by
semicolons, then they could specify a function to processcol3:bulk_import.upload_file("part","msgpack",input_data,dtypes={"col2":"str"},converters={"col3",lambdax:x.split(";")},)which would produce:1575454204,"a","0001",["a","b","c"]1575454204,"b","0002",["d","e","f"]DevelopmentRunning testsRun tests.$pythonsetup.pytestRunning tests (tox)You can run tests against all supported Python versions. I’d recommend you to installpyenvto manage Pythons.$pyenvshellsystem$forversionin$(cat.python-version);do[-d"$(pyenvroot)/versions/${version}"]||pyenvinstall"${version}";done$pyenvshell--unsetInstalltox.$pipinstalltoxThen, runtox.$toxReleaseRelease to PyPI. Ensure you installed twine.$pythonsetup.pybdist_wheelsdist$twineuploaddist/*LicenseApache Software License, Version 2.0