Package: apache-iotdb
Apache IoTDBApache IoTDB (Database for Internet of Things) is an IoT native database with high performance for
data management and analysis, deployable on the edge and the cloud. Due to its light-weight
architecture, high performance and rich feature set together with its deep integration with
Apache Hadoop, Spark and Flink, Apache IoTDB can meet the requirements of massive data storage,
high-speed data ingestion and complex data analysis in the IoT industrial fields.Python Native APIRequirementsYou have to install thrift (>=0.13) before using the package.How to use (Example)First, download the latest package:pip3 install apache-iotdbNotice: If you are installing Python API v0.13.0, DO NOT install bypip install apache-iotdb==0.13.0, usepip install apache-iotdb==0.13.0.post1instead!You can get an example of using the package to read and write data at here:ExampleAn example of aligned timeseries:Aligned Timeseries Session Example(you need to addimport iotdbin the head of the file)Or:fromiotdb.SessionimportSessionip="127.0.0.1"port_="6667"username_="root"password_="root"session=Session(ip,port_,username_,password_)session.open(False)zone=session.get_time_zone()session.close()InitializationInitialize a Sessionsession=Session(ip,port_,username_,password_,fetch_size=1024,zone_id="UTC+8")Open a session, with a parameter to specify whether to enable RPC compressionsession.open(enable_rpc_compression=False)Notice: this RPC compression status of client must comply with that of IoTDB serverClose a Sessionsession.close()Data Definition Interface (DDL Interface)DATABASE ManagementCREATE DATABASEsession.set_storage_group(group_name)Delete one or several databasessession.delete_storage_group(group_name)session.delete_storage_groups(group_name_lst)Timeseries ManagementCreate one or multiple timeseriessession.create_time_series(ts_path,data_type,encoding,compressor,props=None,tags=None,attributes=None,alias=None)session.create_multi_time_series(ts_path_lst,data_type_lst,encoding_lst,compressor_lst,props_lst=None,tags_lst=None,attributes_lst=None,alias_lst=None)Create aligned timeseriessession.create_aligned_time_series(device_id,measurements_lst,data_type_lst,encoding_lst,compressor_lst)Attention: Alias of measurements arenot supportedcurrently.Delete one or several timeseriessession.delete_time_series(paths_list)Check whether the specific timeseries existssession.check_time_series_exists(path)Data Manipulation Interface (DML Interface)InsertIt is recommended to use insertTablet to help improve write efficiency.Insert a Tablet，which is multiple rows of a device, each row has the same measurementsBetter Write PerformanceSupport null values: fill the null value with any value, and then mark the null value via BitMap (from v0.13)We have two implementations of Tablet in Python API.Normal Tabletvalues_=[[False,10,11,1.1,10011.1,"test01"],[True,100,11111,1.25,101.0,"test02"],[False,100,1,188.1,688.25,"test03"],[True,0,0,0,6.25,"test04"],]timestamps_=[1,2,3,4]tablet_=Tablet(device_id,measurements_,data_types_,values_,timestamps_)session.insert_tablet(tablet_)values_=[[None,10,11,1.1,10011.1,"test01"],[True,None,11111,1.25,101.0,"test02"],[False,100,None,188.1,688.25,"test03"],[True,0,0,0,None,None],]timestamps_=[16,17,18,19]tablet_=Tablet(device_id,measurements_,data_types_,values_,timestamps_)session.insert_tablet(tablet_)Numpy TabletComparing with Tablet, Numpy Tablet is usingnumpy.ndarrayto record data.
With less memory footprint and time cost of serialization, the insert performance will be better.Noticetime and value columns in Tablet are ndarray.recommended to use the specific dtypes to each ndarray, see the example below
(if not, the default dtypes are also ok).importnumpyasnpdata_types_=[TSDataType.BOOLEAN,TSDataType.INT32,TSDataType.INT64,TSDataType.FLOAT,TSDataType.DOUBLE,TSDataType.TEXT,]np_values_=[np.array([False,True,False,True],TSDataType.BOOLEAN.np_dtype()),np.array([10,100,100,0],TSDataType.INT32.np_dtype()),np.array([11,11111,1,0],TSDataType.INT64.np_dtype()),np.array([1.1,1.25,188.1,0],TSDataType.FLOAT.np_dtype()),np.array([10011.1,101.0,688.25,6.25],TSDataType.DOUBLE.np_dtype()),np.array(["test01","test02","test03","test04"],TSDataType.TEXT.np_dtype()),]np_timestamps_=np.array([1,2,3,4],TSDataType.INT64.np_dtype())np_tablet_=NumpyTablet(device_id,measurements_,data_types_,np_values_,np_timestamps_)session.insert_tablet(np_tablet_)# insert one numpy tablet with none into the database.np_values_=[np.array([False,True,False,True],TSDataType.BOOLEAN.np_dtype()),np.array([10,100,100,0],TSDataType.INT32.np_dtype()),np.array([11,11111,1,0],TSDataType.INT64.np_dtype()),np.array([1.1,1.25,188.1,0],TSDataType.FLOAT.np_dtype()),np.array([10011.1,101.0,688.25,6.25],TSDataType.DOUBLE.np_dtype()),np.array(["test01","test02","test03","test04"],TSDataType.TEXT.np_dtype()),]np_timestamps_=np.array([98,99,100,101],TSDataType.INT64.np_dtype())np_bitmaps_=[]foriinrange(len(measurements_)):np_bitmaps_.append(BitMap(len(np_timestamps_)))np_bitmaps_[0].mark(0)np_bitmaps_[1].mark(1)np_bitmaps_[2].mark(2)np_bitmaps_[4].mark(3)np_bitmaps_[5].mark(3)np_tablet_with_none=NumpyTablet(device_id,measurements_,data_types_,np_values_,np_timestamps_,np_bitmaps_)session.insert_tablet(np_tablet_with_none)Insert multiple Tabletssession.insert_tablets(tablet_lst)Insert a Recordsession.insert_record(device_id,timestamp,measurements_,data_types_,values_)Insert multiple Recordssession.insert_records(device_ids_,time_list_,measurements_list_,data_type_list_,values_list_)Insert multiple Records that belong to the same device.
With type info the server has no need to do type inference, which leads a better performancesession.insert_records_of_one_device(device_id,time_list,measurements_list,data_types_list,values_list)Insert with type inferenceWhen the data is of String type, we can use the following interface to perform type inference based on the value of the value itself. For example, if value is "true" , it can be automatically inferred to be a boolean type. If value is "3.2" , it can be automatically inferred as a flout type. Without type information, server has to do type inference, which may cost some time.Insert a Record, which contains multiple measurement value of a device at a timestampsession.insert_str_record(device_id,timestamp,measurements,string_values)Insert of Aligned TimeseriesThe Insert of aligned timeseries uses interfaces like insert_aligned_XXX, and others are similar to the above interfaces:insert_aligned_recordinsert_aligned_recordsinsert_aligned_records_of_one_deviceinsert_aligned_tabletinsert_aligned_tabletsIoTDB-SQL InterfaceExecute query statementsession.execute_query_statement(sql)Execute non query statementsession.execute_non_query_statement(sql)Execute statementsession.execute_statement(sql)Device TemplateCreate Device TemplateThe step for creating a metadata template is as followsCreate the template classAdding child Node，InternalNode and MeasurementNode can be choseExecute create device template functiontemplate=Template(name=template_name,share_time=True)i_node_gps=InternalNode(name="GPS",share_time=False)i_node_v=InternalNode(name="vehicle",share_time=True)m_node_x=MeasurementNode("x",TSDataType.FLOAT,TSEncoding.RLE,Compressor.SNAPPY)i_node_gps.add_child(m_node_x)i_node_v.add_child(m_node_x)template.add_template(i_node_gps)template.add_template(i_node_v)template.add_template(m_node_x)session.create_schema_template(template)Modify Device Template nodesModify nodes in a template, the template must be already created. These are functions that add or delete some measurement nodes.add node in templatesession.add_measurements_in_template(template_name,measurements_path,data_types,encodings,compressors,is_aligned)delete node in templatesession.delete_node_in_template(template_name,path)Set Device Templatesession.set_schema_template(template_name,prefix_path)Uset Device Templatesession.unset_schema_template(template_name,prefix_path)Show Device TemplateShow all device templatessession.show_all_templates()Count all nodes in templatessession.count_measurements_in_template(template_name)Judge whether the path is measurement or not in templates, This measurement must be in the templatesession.count_measurements_in_template(template_name,path)Judge whether the path is exist or not in templates, This path may not belong to the templatesession.is_path_exist_in_template(template_name,path)Show nodes under in device templatesession.show_measurements_in_template(template_name)Show the path prefix where a device template is setsession.show_paths_template_set_on(template_name)Show the path prefix where a device template is used (i.e. the time series has been created)session.show_paths_template_using_on(template_name)Drop Device TemplateDelete an existing metadata template，dropping an already set template is not supportedsession.drop_schema_template("template_python")Pandas SupportTo easily transform a query result to aPandas Dataframethe SessionDataSet has a method.todf()which consumes the dataset and transforms it to a pandas dataframe.Example:fromiotdb.SessionimportSessionip="127.0.0.1"port_="6667"username_="root"password_="root"session=Session(ip,port_,username_,password_)session.open(False)result=session.execute_query_statement("SELECT * FROM root.*")# Transform to Pandas Datasetdf=result.todf()session.close()# Now you can work with the dataframedf=...IoTDB TestcontainerThe Test Support is based on the libtestcontainers(https://testcontainers-python.readthedocs.io/en/latest/index.html) which you need to install in your project if you want to use the feature.To start (and stop) an IoTDB Database in a Docker container simply do:classMyTestCase(unittest.TestCase):deftest_something(self):withIoTDBContainer()asc:session=Session("localhost",c.get_exposed_port(6667),"root","root")session.open(False)result=session.execute_query_statement("SHOW TIMESERIES")print(result)session.close()by default it will load the imageapache/iotdb:latest, if you want a specific version just pass it like e.g.IoTDBContainer("apache/iotdb:0.12.0")to get version0.12.0running.IoTDB DBAPIIoTDB DBAPI implements the Python DB API 2.0 specification (https://peps.python.org/pep-0249/), which defines a common
interface for accessing databases in Python.ExamplesInitializationThe initialized parameters are consistent with the session part (except for the sqlalchemy_mode).fromiotdb.dbapiimportconnectip="127.0.0.1"port_="6667"username_="root"password_="root"conn=connect(ip,port_,username_,password_,fetch_size=1024,zone_id="UTC+8",sqlalchemy_mode=False)cursor=conn.cursor()simple SQL statement executioncursor.execute("SELECT * FROM root.*")forrowincursor.fetchall():print(row)execute SQL with parameterIoTDB DBAPI supports pyformat style parameterscursor.execute("SELECT * FROM root.* WHERE time <%(time)s",{"time":"2017-11-01T00:08:00.000"})forrowincursor.fetchall():print(row)execute SQL with parameter sequencesseq_of_parameters=[{"timestamp":1,"temperature":1},{"timestamp":2,"temperature":2},{"timestamp":3,"temperature":3},{"timestamp":4,"temperature":4},{"timestamp":5,"temperature":5},]sql="insert into root.cursor(timestamp,temperature) values(%(timestamp)s,%(temperature)s)"cursor.executemany(sql,seq_of_parameters)close the connection and cursorcursor.close()conn.close()IoTDB SQLAlchemy Dialect (Experimental)The SQLAlchemy dialect of IoTDB is written to adapt to Apache Superset.
This part is still being improved.
Please do not use it in the production environment!Mapping of the metadataThe data model used by SQLAlchemy is a relational data model, which describes the relationships between different entities through tables.
While the data model of IoTDB is a hierarchical data model, which organizes the data through a tree structure.
In order to adapt IoTDB to the dialect of SQLAlchemy, the original data model in IoTDB needs to be reorganized.
Converting the data model of IoTDB into the data model of SQLAlchemy.The metadata in the IoTDB are：DatabasePathEntityMeasurementThe metadata in the SQLAlchemy are：SchemaTableColumnThe mapping relationship between them is：The metadata in the SQLAlchemyThe metadata in the IoTDBSchemaDatabaseTablePath ( from database to entity ) + EntityColumnMeasurementThe following figure shows the relationship between the two more intuitively:Data type mappingdata type in IoTDBdata type in SQLAlchemyBOOLEANBooleanINT32IntegerINT64BigIntegerFLOATFloatDOUBLEFloatTEXTTextLONGBigIntegerExampleexecute statementfromsqlalchemyimportcreate_engineengine=create_engine("iotdb://root:root@127.0.0.1:6667")connect=engine.connect()result=connect.execute("SELECT ** FROM root")forrowinresult.fetchall():print(row)ORM (now only simple queries are supported)fromsqlalchemyimportcreate_engine,Column,Float,BigInteger,MetaDatafromsqlalchemy.ext.declarativeimportdeclarative_basefromsqlalchemy.ormimportsessionmakermetadata=MetaData(schema='root.factory')Base=declarative_base(metadata=metadata)classDevice(Base):__tablename__="room2.device1"Time=Column(BigInteger,primary_key=True)temperature=Column(Float)status=Column(Float)engine=create_engine("iotdb://root:root@127.0.0.1:6667")DbSession=sessionmaker(bind=engine)session=DbSession()res=session.query(Device.status).filter(Device.temperature>1)forrowinres:print(row)DevelopersIntroductionThis is an example of how to connect to IoTDB with python, using the thrift rpc interfaces. Things are almost the same on Windows or Linux, but pay attention to the difference like path separator.PrerequisitesPython3.7 or later is preferred.You have to install Thrift (0.11.0 or later) to compile our thrift file into python code. Below is the official tutorial of installation, eventually, you should have a thrift executable.http://thrift.apache.org/docs/install/Before starting you need to installrequirements_dev.txtin your python environment, e.g. by callingpipinstall-rrequirements_dev.txtCompile the thrift library and DebugIn the root of IoTDB's source code folder,  runmvn clean generate-sources -pl client-py -am.This will automatically delete and repopulate the folderiotdb/thriftwith the generated thrift files.
This folder is ignored from git and shouldnever be pushed to git!NoticeDo not uploadiotdb/thriftto the git repo.Session Client & ExampleWe packed up the Thrift interface inclient-py/src/iotdb/Session.py(similar with its Java counterpart), also provided an example fileclient-py/src/SessionExample.pyof how to use the session module. please read it carefully.Or, another simple example:fromiotdb.SessionimportSessionip="127.0.0.1"port_="6667"username_="root"password_="root"session=Session(ip,port_,username_,password_)session.open(False)zone=session.get_time_zone()session.close()TestsPlease add your custom tests intestsfolder.To run all defined tests just typepytest .in the root folder.NoticeSome tests need docker to be started on your system as a test instance is started in a docker container usingtestcontainers.Futher Toolsblackandflake8are installed for autoformatting and linting.
Both can be run byblack .orflake8 .respectively.ReleasingTo do a release just ensure that you have the right set of generated thrift files.
Then run linting and auto-formatting.
Then, ensure that all tests work (viapytest .).
Then you are good to go to do a release!Preparing your environmentFirst, install all necessary dev dependencies viapip install -r requirements_dev.txt.Doing the ReleaseThere is a convenient scriptrelease.shto do all steps for a release.
Namely, these areRemove all transient directories from last release (if exists)(Re-)generate all generated sources via mvnRun Linting (flake8)Run Tests via pytestBuildRelease to pypi