Package: arrow-odbc
arrow-odbc-pyFill Apache Arrow arrays from ODBC data sources. This package is build on top of thepyarrowPython package andarrow-odbcRust crate and enables you to read the data of an ODBC data source as sequence of Apache Arrow record batches.Fast. Makes efficient use of ODBC bulk reads and writes, to lower IO overhead.Flexible. Query any ODBC data source you have a driver for. MySQL, MS SQL, Excel, ...Portable. Easy to install and update dependencies. No binary dependency to specific implemenations of Python interpreter, Arrow or ODBC driver manager.About ArrowApache Arrowdefines a language-independent columnar memory format for flat and hierarchical data, organized for efficient analytic operations on modern hardware like CPUs and GPUs. The Arrow memory format also supports zero-copy reads for lightning-fast data access without serialization overhead.About ODBCODBC(Open DataBase Connectivity) is a standard which enables you to access data from a wide variaty of data sources using SQL.UsageQueryfromarrow_odbcimportread_arrow_batches_from_odbcconnection_string="Driver={ODBC Driver 17 for SQL Server};Server=localhost;"reader=read_arrow_batches_from_odbc(query=f"SELECT * FROM MyTable WHERE a=?",connection_string=connection_string,parameters=["I'm a positional query parameter"],user="SA",password="My@Test@Password",)# Trade memory for speed. For the price of an additional transit buffer and a native system thread# we fetch batches now concurrent to our application logic. Just remove this line, if you want to# fetch sequentially in your main application thread.reader.fetch_concurrently()forbatchinreader:# Process arrow batchesdf=batch.to_pandas()# ...Insertfromarrow_odbcimportinsert_into_tableimportpyarrowaspaimportpandasdefdataframe_to_table(df):table=pa.Table.from_pandas(df)reader=pa.RecordBatchReader.from_batches(table.schema,table.to_batches())insert_into_table(connection_string=connection_string,user="SA",password="My@Test@Password",chunk_size=1000,table="MyTable",reader=reader,)InstallationInstalling ODBC driver managerThe provided wheels dynamically link against the driver manager, which must be provided by the system.WindowsNothing to do. ODBC driver manager is preinstalled.Ubuntusudoapt-getinstallunixodbc-devOS-XYou can use homebrew to install UnixODBCbrewinstallunixodbcInstalling the wheelThis package has been designed to be easily deployable, so it provides a prebuild many linux wheel which is independent of the specific version of your Python interpreter and the specific Arrow Version you want to use. It will dynamically link against the ODBC driver manager provided by your system.Wheels have been uploaded toPyPiand can be installed using pip. The wheel (including the manylinux wheel) will link against the your system ODBC driver manager at runtime. If there are no prebuild wheels for your platform, you can build the wheel from source. For this the rust toolchain must be installed.pipinstallarrow-odbcarrow-odbcutilizescffiand the Arrow C-Interface to glue Rust and Python code together. Therefore the wheel does not need to be build against the precise version either of Python or Arrow.Installing with condacondainstall-cconda-forgearrow-odbcThanks to @timkpaine for maintaining the recipie!Building wheel from sourceThere is no ready made wheel for the platform you want to target? Do not worry, you can probably build it from source.To build from source you need to install the Rust toolchain. Installation instruction can be found here:https://www.rust-lang.org/tools/installInstall ODBC driver manager. See above.Build wheelpython-mpipinstallbuild
python-mbuildBuilding wheel from source on Mac ARMFollowing above instruction on an Mac ARM will lead to the build process erroring out with a message that theodbclibrary can not be found for linkning. This is becausebrewchooses to install the library into a different folder on this platform. One way to fix this is to create a symbolic link.sudoln-s/opt/homebrew/lib/Users/your_user_name/libUsing this addition stepcargofrom the rust build chain is able to find theodbclibrary and to link against it. Alternatively you can install unixODBC from source usingmake.Matching of ODBC to Arrow types then queryingODBCArrowNumeric(p <= 38)Decimal128Decimal(p <= 38, s >= 0)Decimal128IntegerInt32SmallIntInt16RealFloat32Float(p <=24)Float32DoubleFloat64Float(p > 24)Float64DateDate32LongVarbinaryBinaryTimestamp(p = 0)TimestampSecondTimestamp(p: 1..3)TimestampMilliSecondTimestamp(p: 4..6)TimestampMicroSecondTimestamp(p >= 7 )TimestampNanoSecondBigIntInt64TinyIntInt8BitBooleanVarbinaryBinaryBinaryFixedSizedBinaryAll othersUtf8Matching of Arrow to ODBC types then insertingArrowODBCUtf8VarCharDecimal128(p, s = 0)VarChar(p + 1)Decimal128(p, s != 0)VarChar(p + 2)Decimal128(p, s < 0)VarChar(p - s + 1)Decimal256(p, s = 0)VarChar(p + 1)Decimal256(p, s != 0)VarChar(p + 2)Decimal256(p, s < 0)VarChar(p - s + 1)Int8TinyIntInt16SmallIntInt32IntegerInt64BigIntFloat16RealFloat32RealFloat64DoubleTimestamp sTimestamp(7)Timestamp msTimestamp(7)Timestamp usTimestamp(7)Timestamp nsTimestamp(7)Date32DateDate64DateTime32 sTimeTime32 msVarChar(12)Time64 usVarChar(15)Time64 nsVarChar(16)BinaryVarbinaryFixedBinary(l)Varbinary(l)All othersUnsupportedComparision to other Python ODBC bindingspyodbc- General purpose ODBC python bindings. In contrastarrow-odbcis specifically concerned with bulk reads and writes to arrow arrays.turbodbc- Complies with the Python Database API Specification 2.0 (PEP 249) whicharrow-odbcdoes not aim to do. Likearrow-odbcbulk read and writes is the strong point ofturbodbc.turbodbchas more system dependencies, which can make it cumbersome to install if not using conda.turbodbcis build against the C++ implementation of Arrow, which implies it is only compatible with matching version ofpyarrow.