Package: tuplex
Tuplex: Blazing Fast Python Data ScienceWebsiteDocumentationTuplex is a parallel big data processing framework that runs data science pipelines written in Python at the speed of compiled code.
Tuplex has similar Python APIs toApache SparkorDask, but rather than invoking the Python interpreter, Tuplex generates optimized LLVM bytecode for the given pipeline and input data set. Under the hood, Tuplex is based on data-driven compilation and dual-mode processing, two key techniques that make it possible for Tuplex to provide speed comparable to a pipeline written in hand-optimized C++.You can join the discussion on Tuplex on ourGitter communityor read up more on the background of Tuplex in ourSIGMOD'21 paper.Contributions welcome!ContentsExampleQuickstartInstallationDocker imagePypiBuildingMacOS build from sourceUbuntu build from sourceCustomizing the buildLicenseExampleTuplex can be used in python interactive mode, a jupyter notebook or by copying the below code to a file. To try it out, run the following example:fromtupleximport*c=Context()res=c.parallelize([1,2,None,4]).map(lambdax:(x,x*x)).collect()# this prints [(1, 1), (2, 4), (4, 16)]print(res)QuickstartTo try out Tuplex, simply try out the following starter notebooks using Google Colab:NameLinkDescription1. Intro to TuplexGoogle ColabBasic commands to manipulate columns and modify data with user code.2. Working with FilesGoogle ColabLoading and saving files, detecting types.More examples can be foundhere.InstallationTo install Tuplex, you can use a PyPi package for Linux or MacOS(Intel), or a Docker container which will launch a jupyter notebook with Tuplex preinstalled.Dockerdocker run -p 8888:8888 tuplex/tuplex:v0.3.6PyPIpip install tuplexBuildingTuplex is available for MacOS and Linux. The current version has been tested under MacOS 10.13+ and Ubuntu 20.04/22.04 LTS.
To install Tuplex, simply install the dependencies first and then build the package.MacOS build from sourceTo build Tuplex, you need several other packages first which can be easily installed viabrew. If you want to build Tuplex with AWS support, you needmacOS 10.13+. Python 3.9 or earlier requires an older cloudpickle version (1.6.0) whereas Python 3.10+ requires cloudpickle 2.1.0+.brew install llvm@9 boost boost-python3 aws-sdk-cpp pcre2 antlr4-cpp-runtime googletest gflags yaml-cpp celero protobuf libmagic
python3 -m pip install 'cloudpickle<2.0' numpy
python3 setup.py install --userUbuntu build from sourceTo faciliate installing the dependencies for Ubuntu, we do provide two scripts (scripts/ubuntu2004/install_reqs.shfor Ubuntu 20.04, orscripts/ubuntu2204/install_reqs.shfor Ubuntu 22.04). To create an up to date version of Tuplex, simply run./scripts/ubuntu2204/install_reqs.sh
python3 -m pip install cloudpickle numpy
python3 setup.py install --userCustomizing the buildBesides building a pip package, especially for development it may be more useful to invoke cmake directly. To create a development version of Tuplex and work with it like a regular cmake project, go to the foldertuplexand then use the standard workflow to compile the package via cmake (and not the top-level setup.py file):mkdir build
cd build
cmake ..
make -j$(nproc)The python package corresponding to Tuplex can be then found inbuild/dist/pythonwith C++ test executables based on googletest inbuild/dist/bin. If you'd like to use a cmake-compatible IDE like CLion or VSCode you can simply open thetuplex/folder and import theCMakeLists.txtcontained there.To customize the cmake build, the following options are available to be passed via-D<option>=<value>:optionvaluesdescriptionCMAKE_BUILD_TYPERelease(default),Debug,RelWithDebInfo,tsan,asan,ubsanselect compile mode. Tsan/Asan/Ubsan correspond to Google Sanitizers.BUILD_WITH_AWSON(default),OFFbuild with AWS SDK or not. On Ubuntu this will build the Lambda executor.BUILD_WITH_ORCON,OFF(default)build with ORC file format support.BUILD_NATIVEON,OFF(default)build with-march=nativeto target platform architecture.SKIP_AWS_TESTSON(default),OFFskip aws tests, helpful when no AWS credentials/AWS Tuplex chain is setup.GENERATE_PDFSON,OFF(default)output in Debug mode PDF files if graphviz is installed (e.g.,brew install graphviz) for ASTs of UDFs, query plans, ...PYTHON3_VERSION3.8, ...when trying to select a python3 version to build against, use this by specifyingmajor.minor. To specify the python executable, use the options provided bycmake.LLVM_ROOT_DIRe.g./usr/lib/llvm-9specify which LLVM version to useBOOST_DIRe.g./opt/boostspecify which Boost version to use. Note that the python component of boost has to be built against the python version used to build TuplexFor example, to create a debug build which outputs PDFs use the following snippet:cmake -DCMAKE_BUILD_TYPE=Debug -DGENERATE_PDFS=ON ..LicenseTuplex is available under Apache 2.0 License, to cite the paper use:@inproceedings{10.1145/3448016.3457244,author={Spiegelberg, Leonhard and Yesantharao, Rahul and Schwarzkopf, Malte and Kraska, Tim},title={Tuplex: Data Science in Python at Native Code Speed},year={2021},isbn={9781450383431},publisher={Association for Computing Machinery},address={New York, NY, USA},url={https://doi.org/10.1145/3448016.3457244},doi={10.1145/3448016.3457244},booktitle={Proceedings of the 2021 International Conference on Management of Data},pages={1718–1731},numpages={14},location={Virtual Event, China},series={SIGMOD/PODS '21}}(c) 2017-2023 Tuplex contributors