Package: connectorx
ConnectorXLoad data fromto, the fastest way.ConnectorX enables you to load data from databases into Python in the fastest and most memory efficient way.What you need is one line of code:importconnectorxascxcx.read_sql("postgresql://username:password@server:port/database","SELECT * FROM lineitem")Optionally, you can accelerate the data loading using parallelism by specifying a partition column.importconnectorxascxcx.read_sql("postgresql://username:password@server:port/database","SELECT * FROM lineitem",partition_on="l_orderkey",partition_num=10)The function will partition the query byevenlysplitting the specified column to the amount of partitions.
ConnectorX will assign one thread for each partition to load and write data in parallel.
Currently, we support partitioning onnumericalcolumns (cannot contain NULL) forSPJAqueries.Experimental: We are now providing federated query support (PostgreSQL only and do not support partition for now), you can write a single query to join tables from two or more databases! (JRE >= 1.8 is required)importconnectorxascxdb1="postgresql://username1:password1@server1:port1/database1"db2="postgresql://username2:password2@server2:port2/database2"cx.read_sql({"db1":db1,"db2":db2},"SELECT * FROM db1.nation n, db2.region r where n.n_regionkey = r.r_regionkey")Check out more detailed usage and exampleshere. A general introduction of the project can be found in thisblog post.InstallationpipinstallconnectorxCheck outhereto see how to build python wheel from source.PerformanceWe compared different solutions in Python that provides theread_sqlfunction, by loading a 10x TPC-H lineitem table (8.6GB) from Postgres into a DataFrame, with 4 cores parallelism.Time chart, lower is better.Memory consumption chart, lower is better.In conclusion, ConnectorX uses up to3xless memory and21xless time (3xless memory and13xless time compared with Pandas.). More onhere.How does ConnectorX achieve a lightning speed while keeping the memory footprint low?We observe that existing solutions more or less do data copy multiple times when downloading the data.
Additionally, implementing a data intensive application in Python brings additional cost.ConnectorX is written in Rust and follows "zero-copy" principle.
This allows it to make full use of the CPU by becoming cache and branch predictor friendly. Moreover, the architecture of ConnectorX ensures the data will be copied exactly once, directly from the source to the destination.How does ConnectorX download the data?Upon receiving the query, e.g.SELECT * FROM lineitem, ConnectorX will first issue aLIMIT 1querySELECT * FROM lineitem LIMIT 1to get the schema of the result set.Then, ifpartition_onis specified, ConnectorX will issueSELECT MIN($partition_on), MAX($partition_on) FROM (SELECT * FROM lineitem)to know the range of the partition column.
After that, the original query is split into partitions based on the min/max information, e.g.SELECT * FROM (SELECT * FROM lineitem) WHERE $partition_on > 0 AND $partition_on < 10000.
ConnectorX will then run a count query to get the partition size (e.g.SELECT COUNT(*) FROM (SELECT * FROM lineitem) WHERE $partition_on > 0 AND $partition_on < 10000). If the partition
is not specified, the count query will beSELECT COUNT(*) FROM (SELECT * FROM lineitem).Finally, ConnectorX will use the schema info as well as the count info to allocate memory and download data by executing the queries normally.Once the downloading begins, there will be one thread for each partition so that the data are downloaded in parallel at the partition level. The thread will issue the query of the corresponding
partition to the database and then write the returned data to the destination row-wise or column-wise (depends on the database) in a streaming fashion.Supported Sources & DestinationsExample connection string, supported protocols and data types for each data source can be foundhere.For more planned data sources, please check out ourdiscussion.SourcesPostgresMysqlMariadb (through mysql protocol)SqliteRedshift (through postgres protocol)Clickhouse (through mysql protocol)SQL ServerAzure SQL Database (through mssql protocol)OracleBig QueryODBC (WIP)...DestinationsPandasPyArrowModin (through Pandas)Dask (through Pandas)Polars (through PyArrow)DocumentationDoc:https://sfu-db.github.io/connector-x/intro.htmlRust docs:stablenightlyNext PlanCheckout ourdiscussionto participate in deciding our next plan!Historical Benchmark Resultshttps://sfu-db.github.io/connector-x/dev/bench/Developer's GuidePlease seeDeveloper's Guidefor information about developing ConnectorX.SupportsYou are always welcomed to:Ask questions & propose new ideas in our githubdiscussion.Ask questions in stackoverflow. Make sure to have #connectorx attached.Organizations and Projects using ConnectorXTo add your project/organization here, reply our posthereCiting ConnectorXIf you use ConnectorX, please consider citing the following paper:Xiaoying Wang, Weiyuan Wu, Jinze Wu, Yizhou Chen, Nick Zrymiak, Changbo Qu, Lampros Flokas, George Chow, Jiannan Wang, Tianzheng Wang, Eugene Wu, Qingqing Zhou.ConnectorX: Accelerating Data Loading From Databases to Dataframes.VLDB 2022.BibTeX entry:@article{connectorx2022,author={Xiaoying Wang and Weiyuan Wu and Jinze Wu and Yizhou Chen and Nick Zrymiak and Changbo Qu and Lampros Flokas and George Chow and Jiannan Wang and Tianzheng Wang and Eugene Wu and Qingqing Zhou},title={ConnectorX: Accelerating Data Loading From Databases to Dataframes},journal={Proc. {VLDB} Endow.},volume={15},number={11},pages={2994--3003},year={2022},url={https://www.vldb.org/pvldb/vol15/p2994-wang.pdf},}