Package: proxystore
ProxyStoreProxyStore is a library that facilitates efficient data management in distributed Python applications.
At the core of ProxyStore is theproxyobject which acts as a transparent reference to an object living in a global object store.
This pass-by-reference interface with just-in-time object resolution works across processes, machines, and sites, and enables data producers to change the low-level communication method dynamically without altering application code or behavior.ProxyStore accelerates the development of dynamic task-based workflows, serverless applications, and more by (1) decoupling control flow from data flow, (2) abstracting low-level communication mechanisms, and (3) eliminating the need for shims, wrapper functions, and boilerplate code.ProxyStore supports a diverse set of programming paradigms:Task-based WorkflowsFunction-as-a-Service/Serverless ApplicationsDistributed FuturesBulk Data Streamingand more!ProxyStore provides support for many third-party mediated communication methods
out-of-the-box includingDAOS,Globus Transfer,Kafka,KeyDB, andRedis.
Custom communication methods built onMochi,UCX,WebRTC, andZeroMQare provided for high-performance and peer-to-peer applications.Read more about ProxyStore's conceptshere.
Complete documentation for ProxyStore is available atdocs.proxystore.dev.InstallationThe base ProxyStore package can be installed withpip.pipinstallproxystoreMany features require dependencies that are not installed by default but can
be enabled via extras installation options such asendpoints,kafka, orredis.Alloptional dependencies can be installed with:pipinstallproxystore[all]This will also install theproxystore-expackage which contains extension and experimental features.
The extensions package can also be installed withpipusingproxystore[extensions]orproxystore-ex.See theInstallationguide
for more information about the available extras installation options.
See theContributingguide
to get started for local development.ExampleGetting started with ProxyStore requires a few lines of code.fromproxystore.connectors.redisimportRedisConnectorfromproxystore.proxyimportProxyfromproxystore.storeimportregister_storefromproxystore.storeimportStorestore=Store('my-store',RedisConnector('localhost',6379))# Store the object and get a proxy. The proxy acts# like a reference to the object.data=MyDataType(...)proxy=store.proxy(data)assertisinstance(proxy,Proxy)defmy_function(x:MyDataType)->...:# x is resolved my-store on first use transparently to the# function. Then x behaves as an instance of MyDataType.assertisinstance(x,MyDataType)my_function(proxy)# SucceedsCheck out theGet Startedguide to learn more!CitationIf you use ProxyStore or any of this code in your work, please cite ourSC '23 paper.@inproceedings{pauloski2023proxystore,author={Pauloski, J. Gregory and Hayot-Sasson, Valerie and Ward, Logan and Hudson, Nathaniel and Sabino, Charlie and Baughman, Matt and Chard, Kyle and Foster, Ian},title={{Accelerating Communications in Federated Applications with Transparent Object Proxies}},address={New York, NY, USA},articleno={59},booktitle={Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis},doi={10.1145/3581784.3607047},isbn={9798400701092},location={Denver, CO, USA},numpages={15},publisher={Association for Computing Machinery},series={SC '23},url={https://doi.org/10.1145/3581784.3607047},year={2023}}