Package: pytd
pytdprovides user-friendly interfaces to Treasure Data’sREST
APIs,Presto
query
engine,
andPlazma primary
storage.The seamless connection allows your Python code to efficiently
read/write a large volume of data from/to Treasure Data. Eventually,
pytd makes your day-to-day data analytics work more productive.InstallationpipinstallpytdUsageDocumentationSample usage on Google
ColaboratorySet yourAPI
keyandendpointto the environment variables,TD_API_KEYandTD_API_SERVER,
respectively, and create a client instance:importpytdclient=pytd.Client(database='sample_datasets')# or, hard-code your API key, endpoint, and/or query engine:# >>> pytd.Client(apikey='1/XXX', endpoint='https://api.treasuredata.com/', database='sample_datasets', default_engine='presto')Query in Treasure DataIssue Presto query and retrieve the result:client.query('select symbol, count(1) as cnt from nasdaq group by 1 order by 1')# {'columns': ['symbol', 'cnt'], 'data': [['AAIT', 590], ['AAL', 82], ['AAME', 9252], ..., ['ZUMZ', 2364]]}In case of Hive:client.query('select hivemall_version()',engine='hive')# {'columns': ['_c0'], 'data': [['0.6.0-SNAPSHOT-201901-r01']]} (as of Feb, 2019)It is also possible to explicitly initializepytd.Clientfor Hive:client_hive=pytd.Client(database='sample_datasets',default_engine='hive')client_hive.query('select hivemall_version()')Write data to Treasure DataData represented aspandas.DataFramecan be written to Treasure Data
as follows:importpandasaspddf=pd.DataFrame(data={'col1':[1,2],'col2':[3,10]})client.load_table_from_dataframe(df,'takuti.foo',writer='bulk_import',if_exists='overwrite')For thewriteroption, pytd supports three different ways to ingest
data to Treasure Data:Bulk Import API:bulk_import(default)Convert data into a CSV file and upload in the batch fashion.Presto INSERT INTO query:insert_intoInsert every single row inDataFrameby issuing an INSERT INTO
query through the Presto query engine.Recommended only for a small volume of data.td-spark:sparkLocal customized Spark instance directly writesDataFrameto
Treasure Data’s primary storage system.Characteristics of each of these methods can be summarized as follows:bulk_importinsert_intosparkScalable against data volume✓✓Write performance for larger data✓Memory efficient✓✓Disk efficient✓Minimal package dependency✓✓Enabling Spark WriterSince td-spark gives special access to the main storage system viaPySpark,
follow the instructions below:Contactsupport@treasuredata.comto activate the permission to your
Treasure Data account. Note that the underlying component, Plazma Public
API, limits its free tier at 100GB Read and 100TB Write.Install pytd with[spark]option if you use the third option:pip install pytd[spark]If you want to use existing td-spark JAR file, creatingSparkWriterwithtd_spark_pathoption would be helpful.frompytd.writerimportSparkWriterwriter=SparkWriter(td_spark_path='/path/to/td-spark-assembly.jar')client.load_table_from_dataframe(df,'mydb.bar',writer=writer,if_exists='overwrite')Comparison between pytd, td-client-python, and pandas-tdTreasure Data offers three different Python clients on GitHub, and the following list summarizes their characteristics.td-client-pythonBasic REST API wrapper.Similar functionalities to td-client-{ruby,java,node,go}.The capability is limited bywhat Treasure Data REST API can do.pytdAccess to Plazma via td-spark as introduced above.Efficient connection to Presto based onpresto-python-client.Multiple data ingestion methods and a variety of utility functions.pandas-td(deprecated)Old tool optimized forpandasandJupyter Notebook.pytdoffers its compatible function set (see below for the detail).An optimal choice of package depends on your specific use case, but common guidelines can be listed as follows:Use td-client-python if you want to executebasic CRUD operationsfrom Python applications.Usepytdfor (1)analytical purposerelying on pandas and Jupyter Notebook, and (2) achievingmore efficient data accessat ease.Do not use pandas-td. If you are using pandas-td, replace the code with pytd based on the following guidance as soon as possible.How to replace pandas-tdpytdofferspandas-td-compatible
functions that provide the same functionalities more efficiently. If you
are still using pandas-td, we recommend you to switch topytdas
follows.First, install the package from PyPI:pipinstallpytd# or, `pip install pytd[spark]` if you wish to use `to_td`Next, make the following modifications on the import statements.Before:importpandas_tdastdIn[1]:%%load_extpandas_td.ipythonAfter:importpytd.pandas_tdastdIn[1]:%%load_extpytd.pandas_td.ipythonConsequently, allpandas_tdcode should keep running correctly withpytd. Report an issue fromhereif you
noticed any incompatible behaviors.