Package: pathway
Getting Started|Example|Performance|Deployment|Resources|Documentation|Blog|Get HelpPathwayPathwayis an open framework for high-throughput and low-latency real-time data processing. It is used to create Python code which seamlessly combines batch processing, streaming, and real-time API's for LLM apps. Pathway's distributed runtime (🦀-🐍) provides fresh results of your data pipelines whenever new inputs and requests are received.In the first place, Pathway was designed to be a life-saver (or at least a time-saver) for Python developers and ML/AI engineers faced with live data sources, where you need to react quickly to fresh data. Still, Pathway is a powerful tool that can be used for a lot of things. If you want to do streaming in Python, build an AI data pipeline, or if you are looking for your next Python data processing framework, keep reading.Pathway provides a high-level programming interface in Python for defining data transformations, aggregations, and other operations on data streams.
With Pathway, you can effortlessly design and deploy sophisticated data workflows that efficiently handle high volumes of data in real time.Pathway is interoperable with various data sources and sinks such as Kafka, CSV files, SQL/noSQL databases, and REST API's, allowing you to connect and process data from different storage systems.Typical use-cases of Pathway include realtime data processing, ETL (Extract, Transform, Load) pipelines, data analytics, monitoring, anomaly detection, and recommendation. Pathway can also independently provide the backbone of a light LLMops stack forreal-time LLM applications.In Pathway, data is represented in the form of Tables. Live data streams are also treated as Tables. The library provides a rich set of operations like filtering, joining, grouping, and windowing.For any questions, you will find the community and team behind the projecton Discord.Getting startedInstallationPathway requires Python 3.10 or above.You can install the current release of Pathway usingpip:$ pip install -U pathway⚠️ Pathway is available on MacOS and Linux. Users of other systems should run Pathway on a Virtual Machine.Running Pathway locallyTo use Pathway, you only need to import it:importpathwayaspwNow, you can easily create your processing pipeline, and let Pathway handle the updates. Once your pipeline is created, you can launch the computation on streaming data with a one-line command:pw.run()You can then run your Pathway project (say,main.py) just like a normal Python script:$ python main.py. Alternatively, use the pathway'ish version:$ pathway spawn python main.pyPathway natively supports multithreading.
To launch your application with 3 threads, you can do as follows:$ pathway spawn --threads 3 python main.pyTo jumpstart a Pathway project, you can use ourcookiecutter template.Exampleimportpathwayaspw# Using the `demo` module to create a data streamtable=pw.demo.range_stream(nb_rows=50)# Storing the stream into a CSV filepw.io.csv.write(table,"output_table.csv")# Summing all the values in a new tablesum_table=table.reduce(sum=pw.reducers.sum(pw.this.value))# Storing the sum (which is a stream) in another CSV filepw.io.csv.write(sum_table,"sum_table.csv")# Now that the pipeline is built, the computation is startedpw.run()Run this examplein Google Colab!You can find more exampleshere.DeploymentDo you feel limited by a local run?
If you want to scale your Pathway application, you may be interested in our Pathway for Enterprise.
Pathway for Enterprise is specially tailored towards end-to-end data processing and real time intelligent analytics.
It scales using distributed computing on the cloud and supports Kubernetes deployment.You can learn more about the features of Pathway for Enterprise on ourwebsite.If you are interested, don't hesitate tocontact usto learn more.Monitoring PathwayPathway comes with a monitoring dashboard that allows you to keep track of the number of messages sent by each connector and the latency of the system. The dashboard also includes log messages.This dashboard is enabled by default; you can disable it by passingmonitoring_level = pathway.MonitoringLevel.NONEtopathway.run().In addition to Pathway's built-in dashboard, you canuse Prometheusto monitor your Pathway application.ResourcesSee also:📖Pathway Documentationwebpage (including API Docs).Videos about Pathway▶️ Building an LLM Application without a vector database- byJan Chorowski(7min 56s)▶️ Linear regression on a Kafka Stream- byRichard Pelgrim(7min 53s)GuidesCore concepts of PathwayBasic operationsJoinsGroupbyWindowbyTransformer classesInput and output connectorsComing from pandasAPI docsTroubleshootingTutorialsLinear regression on a Kafka Stream(video)Joins:Interval joinsWindow joinsASOF joinsConnectors:CSV connectorsDatabase connectorsKafka connectorsCustom Python connectorSwitching from Kafka to RedpandaMonitoring Pathway with PrometheusTime between events in a multi-topic event streamShowcasesRealtime Twitter Analysis AppRealtime classification with Nearest NeighborsRealtime Fuzzy joinsExternal and community contentReal-time linear regression (Data Engineering Weekly)Realtime server logs monitoring (Data Engineering Weekly)Data enrichment with fuzzy joins (Data Engineering Weekly)▶️ How to do Realtime Twitter Sentiment Analysis in Python (video)If you would like to share with us some Pathway-related content, please give an admin a shout onDiscord.Manul conventionsManuls (aka Pallas's Cats)are creatures with fascinating habits. As a tribute to them, we usually readpw, one of the most frequent tokens in Pathway code, as:"paw".PerformancePathway is made to outperform state-of-the-art technologies designed for streaming and batch data processing tasks, including: Flink, Spark, and Kafka Streaming. It also makes it possible to implement a lot of algorithms/UDF's in streaming mode which are not readily supported by other streaming frameworks (especially: temporal joins, iterative graph algorithms, machine learning routines).If you are curious, here aresome benchmarks to play with.If you try your own benchmarks, please don't hesitate to let us know. We investigate situations in which Pathway is underperforming on par with bugs (i.e., to our knowledge, they shouldn't happen...).Coming soonHere are some features we plan to incorporate in the near future:Enhanced monitoring, observability, and data drift detection (integrates with Grafana visualization and other dashboarding tools).New connectors: interoperability with Delta Lake and Snowflake data sources.Easier connection setup for MongoDB.More performant garbage collection.DependenciesPathway is made to run in a "clean" Linux/MacOS + Python environment. When installing the pathway package withpip(from a wheel), you are likely to encounter a small number of Python package dependencies, such as sqlglot (used in the SQL API) and python-sat (useful for resolving dependencies during compilation). All necessary Rust crates are pre-built; the Rust compiler is not required to install Pathway, unless building from sources. A modified version of Timely/Differential Dataflow (which provides a dataflow assembly layer) is part of this repo.LicensePathway is distributed on aBSL 1.1 Licensewhich allows for unlimited non-commercial use, as well as use of the Pathway packagefor most commercial purposes, free of charge. Code in this repository automatically converts to Open Source (Apache 2.0 License) after 4 years. Somepublic reposwhich are complementary to this one (examples, libraries, connectors, etc.) are licensed as Open Source, under the MIT license.Contribution guidelinesIf you develop a library or connector which you would like to integrate with this repo, we suggest releasing it first as a separate repo on a MIT/Apache 2.0 license.For all concerns regarding core Pathway functionalities, Issues are encouraged. For further information, don't hesitate to engage with Pathway'sDiscord community.Get HelpIf you have any questions, issues, or just want to chat about Pathway, we're here to help! Feel free to:Check out thedocumentationfor detailed information.Open an issue on GitHubif you encounter any bugs or have feature requests.Join us onDiscordto connect with other users and get support.Reach out to us via email atcontact@pathway.com.Our team is always happy to help you and ensure that you get the most out of Pathway.
If you would like to better understand how best to use Pathway in your project, please don't hesitate to reach out to us.