Package: pylance
Python bindings for Lance Data Format:warning:Under heavy developmentLance is a new columnar data format for data science and machine learningWhy you should use LanceIs order of magnitude faster than parquet for point queries and nested data structures common to DS/MLComes with a fast vector index that delivers sub-millisecond nearest neighbors search performanceIs automatically versioned and supports lineage and time-travel for full reproducibilityIntegrated with duckdb/pandas/polars already. Easily convert from/to parquet in 2 lines of codeQuick startInstallationpipinstallpylanceMake sure you have a recent version of pandas (1.5+), pyarrow (10.0+), and DuckDB (0.7.0+)Converting to Lanceimportlanceimportpandasaspdimportpyarrowaspaimportpyarrow.datasetdf=pd.DataFrame({"a":[5],"b":[10]})uri="/tmp/test.parquet"tbl=pa.Table.from_pandas(df)pa.dataset.write_dataset(tbl,uri,format='parquet')parquet=pa.dataset.dataset(uri,format='parquet')lance.write_dataset(parquet,"/tmp/test.lance")Reading Lance datadataset=lance.dataset("/tmp/test.lance")assertisinstance(dataset,pa.dataset.Dataset)Pandasdf=dataset.to_table().to_pandas()DuckDBimportduckdb# If this segfaults, make sure you have duckdb v0.7+ installedduckdb.query("SELECT * FROM dataset LIMIT 10").to_df()Vector searchDownload the sift1m subsetwgetftp://ftp.irisa.fr/local/texmex/corpus/sift.tar.gz
tar-xzfsift.tar.gzConvert it to Lanceimportlancefromlance.vectorimportvec_to_tableimportnumpyasnpimportstructnvecs=1000000ndims=128withopen("sift/sift_base.fvecs",mode="rb")asfobj:buf=fobj.read()data=np.array(struct.unpack("<128000000f",buf[4:4+4*nvecs*ndims])).reshape((nvecs,ndims))dd=dict(zip(range(nvecs),data))table=vec_to_table(dd)uri="vec_data.lance"sift1m=lance.write_dataset(table,uri,max_rows_per_group=8192,max_rows_per_file=1024*1024)Build the indexsift1m.create_index("vector",index_type="IVF_PQ",num_partitions=256,# IVFnum_sub_vectors=16)# PQSearch the dataset# Get top 10 similar vectorsimportduckdbdataset=lance.dataset(uri)# Sample 100 query vectors. If this segfaults, make sure you have duckdb v0.7+ installedsample=duckdb.query("SELECT vector FROM dataset USING SAMPLE 100").to_df()query_vectors=np.array([np.array(x)forxinsample.vector])# Get nearest neighbors for all of themrs=[dataset.to_table(nearest={"column":"vector","k":10,"q":q})forqinquery_vectors]*More distance metrics, HNSW, and distributed support is on the roadmapPython package detailsInstall from PyPI:pip install pylance# >=0.3.0 is the new rust-based implementation
Install from source:maturin develop(under the/pythondirectory)
Run unit tests:make testRun integration tests:make integtestImport via:import lanceThe python integration is done via pyo3 + custom python code:We make wrapper classes in Rust for Dataset/Scanner/RecordBatchReader that's exposed to python.These are then used by LanceDataset / LanceScanner implementations that extend pyarrow Dataset/Scanner for duckdb compat.Data is delivered via the Arrow C Data InterfaceMotivationWhy do weneeda new format for data science and machine learning?1. Reproducibility is a must-haveVersioning and experimentation support should be built into the dataset instead of requiring multiple tools.It should also be efficient and not require expensive copying everytime you want to create a new version.We call this "Zero copy versioning" in Lance. It makes versioning data easy without increasing storage costs.2. Cloud storage is now the defaultRemote object storage is the default now for data science and machine learning and the performance characteristics of cloud are fundamentally different.Lance format is optimized to be cloud native. Common operations like filter-then-take can be order of magnitude faster
using Lance than Parquet, especially for ML data.3. Vectors must be a first class citizen, not a separate thingThe majority of reasonable scale workflows should not require the added complexity and cost of a
specialized database just to compute vector similarity. Lance integrates optimized vector indices
into a columnar format so no additional infrastructure is required to get low latency top-K similarity search.4. Open standards is a requirementThe DS/ML ecosystem is incredibly rich and datamust beeasily accessible across different languages, tools, and environments.
Lance makes Apache Arrow integration its primary interface, which means conversions to/from is 2 lines of code, your
code does not need to change after conversion, and nothing is locked-up to force you to pay for vendor compute.
We need open-source not fauxpen-source.