Repository: numaproj/numaflow
Pull request #2552: doc: add missing docs
Author: login: vigith
Created: 2025-04-14T18:17:05Z | Merged: 2025-04-16T15:15:04Z

Fixes: #2514

Rearrange docs and add SDKs


Diff:
diff --git a/docs/core-concepts/overview.md b/docs/core-concepts/overview.md
new file mode 100644
index 0000000000..b37a4c86fc
--- /dev/null
+++ b/docs/core-concepts/overview.md
@@ -0,0 +1,50 @@
+# Numaflow Core Concepts Overview
+
+Numaflow is a Kubernetes-native platform for event processing at scale, purpose-built for creating event-driven 
+applications, real-time stream processing pipelines, and serving systems. It provides a set of modular and composable 
+building blocks to help developers process events and serve efficiently in cloud-native environments.
+
+The following sections introduce the core concepts of Numaflow.
+
+* [MonoVertex](#monovertex)
+* [Pipeline](#pipeline)
+* [Serving](#serving)
+
+## MonoVertex
+
+[MonoVertex](./monovertex.md) is a lightweight way to develop event-driven applications in Numaflow. Each MonoVertex contains source, 
+transformer, and sink within a single unit, making it ideal for simple event processing tasks. The entire unit scales as
+one, making it a lightweight option for handling simple event processing patterns.
+
+### Use Cases
+
+- Stateless event transformation  
+- Filtering, mapping, or enriching event data  
+- Microservice-style event handling (e.g., call external APIs per event)  
+- Ingesting from or writing to external systems within a single unit  
+
+
+## Pipeline
+
+[Pipeline](pipeline.md) is designed for developing real-time stream processing pipelines. It allows you to connect multiple vertices — each
+representing a processing step — to handle data. Pipelines can include multiple sources and sinks, enabling integration with
+diverse systems at both the input and output stages.
+
+Each step in a pipeline can perform operations like transformation, routing, reduction, or aggregation, and is independently
+scalable. This makes Pipelines ideal for building robust, modular stream processing applications.
+
+### Use Cases
+- Aggregation and windowing (e.g., sum, count, reduce over time windows)  
+- Routing events based on payload or metadata  
+- Multi-step data processing and enrichment  
+- Scalable data pipelines for analytics and monitoring  
+
+## Serving
+
+[Serving](./serving.md) enables request/response interaction with streaming systems, allowing external clients to send data and 
+receive processed results in real time through interfaces like REST or Server-Sent Events (SSE).
+
+### Use Cases
+- Machine learning inference (e.g., send features, receive predictions)   
+- Event-driven APIs backed by stream processing logic  
+- Asynchronous request/response workflows over SSE or HTTP
diff --git a/docs/user-guide/sdks/compatibility.md b/docs/user-guide/sdks/compatibility.md
new file mode 100644
index 0000000000..2e6d588ee0
--- /dev/null
+++ b/docs/user-guide/sdks/compatibility.md
@@ -0,0 +1,35 @@
+# SDK Compatibility
+
+SDK and Numaflow compatibility version matrix. 
+
+## Python
+
+| Python \ Numaflow  | v1.0.0 | v1.1.0 | v1.2.0 | v1.3.0 | v1.4.0 | v1.5.0 (next) |
+|:-------------------|:------:|:------:|:------:|:------:|:------:|:-------------:|
+| __v0.8.0__         |   ●    |   ●    |   ●    |   ●    |        |               |
+| __v0.9.0__         |        |        |        |        |   ●    |       ○       |
+| __v0.10.0 (next)__ |        |        |        |        |        |       ○       |
+
+## Golang
+
+| Golang  \ Numaflow | v1.0.0 | v1.1.0 | v1.2.0 | v1.3.0 | v1.4.0 | v1.5.0 (next) |
+|:-------------------|:------:|:------:|:------:|:------:|:------:|:-------------:|
+| __v0.8.0__         |   ●    |   ●    |   ●    |   ●    |        |               |
+| __v0.9.0__         |        |        |        |        |   ●    |       ○       |
+| __v0.10.0 (next)__ |        |        |        |        |        |       ○       |
+
+## Java
+
+| Java \ Numaflow    | v1.0.0 | v1.1.0 | v1.2.0 | v1.3.0 | v1.4.0 | v1.5.0 (next) |
+|:-------------------|:------:|:------:|:------:|:------:|:------:|:-------------:|
+| __v0.8.0__         |   ●    |   ●    |   ●    |   ●    |        |               |
+| __v0.9.0__         |        |        |        |        |   ●    |       ○       |
+| __v0.10.0 (next)__ |        |        |        |        |        |       ○       |
+
+## Rust
+
+| Rust \ Numaflow          | v1.0.0 | v1.1.0 | v1.2.0 | v1.3.0 | v1.4.0 | v1.5.0 (next) |
+|:-------------------------|:------:|:------:|:------:|:------:|:------:|:-------------:|
+| __0.1.0__                |        |        |        |   ●    |        |               |
+| __0.2.0__                |        |        |        |        |   ●    |       ○       |
+| __0.3.0 (next)__         |        |        |        |        |        |       ○       |
\ No newline at end of file
diff --git a/docs/user-guide/sdks/features.md b/docs/user-guide/sdks/features.md
new file mode 100644
index 0000000000..61a7404eee
--- /dev/null
+++ b/docs/user-guide/sdks/features.md
@@ -0,0 +1,35 @@
+# SDK Features
+
+The features supported by SDKs. These features are about whether each SDK support writing User Defined Functions.
+
+## Python
+
+|                    | Source | Transformer | Map | Batch Map | Streaming Map | Reduce | Reduce Streaming | Reduce Sessions | Accumulator | Sink |
+|:-------------------|:------:|:-----------:|:---:|:---------:|:-------------:|:------:|:----------------:|:---------------:|:-----------:|:----:|
+| __v0.8.0__         |   ●    |      ●      |  ●  |     ●     |       ●       |   ●    |        ●         |                 |             |  ●   |
+| __v0.9.0__         |   ●    |      ●      |  ●  |     ●     |       ●       |   ●    |        ●         |                 |             |  ●   |
+| __v0.10.0 (next)__ |   ●    |      ●      |  ●  |     ●     |       ●       |   ●    |        ●         |                 |      ○      |  ●   |
+
+## Golang
+
+|                    | Source | Transformer | Map | Batch Map | Streaming Map | Reduce | Reduce Streaming | Reduce Sessions | Accumulator | Sink |
+|:-------------------|:------:|:-----------:|:---:|:---------:|:-------------:|:------:|:----------------:|:---------------:|:-----------:|:----:|
+| __v0.8.0__         |   ●    |      ●      |  ●  |     ●     |       ●       |   ●    |        ●         |        ●        |             |  ●   |
+| __v0.9.0__         |   ●    |      ●      |  ●  |     ●     |       ●       |   ●    |        ●         |        ●        |             |  ●   |
+| __v0.10.0 (next)__ |   ●    |      ●      |  ●  |     ●     |       ●       |   ●    |        ●         |        ●        |      ●      |  ●   |
+
+## Java
+
+|                    | Source | Transformer | Map | Batch Map | Streaming Map | Reduce | Reduce Streaming | Reduce Sessions | Accumulator | Sink |
+|:-------------------|:------:|:-----------:|:---:|:---------:|:-------------:|:------:|:----------------:|:---------------:|:-----------:|:----:|
+| __v0.8.0__         |   ●    |      ●      |  ●  |     ●     |       ●       |   ●    |        ●         |        ●        |             |  ●   |
+| __v0.9.0__         |   ●    |      ●      |  ●  |     ●     |       ●       |   ●    |        ●         |        ●        |             |  ●   |
+| __v0.10.0 (next)__ |   ●    |      ●      |  ●  |     ●     |       ●       |   ●    |        ●         |        ●        |      ●      |  ●   |
+
+## Rust
+
+|                  | Source | Transformer | Map | Batch Map | Streaming Map | Reduce | Reduce Streaming | Reduce Sessions | Accumulator | Sink |
+|:-----------------|:------:|:-----------:|:---:|:---------:|:-------------:|:------:|:----------------:|:---------------:|:-----------:|:----:|
+| __0.1.0__        |   ●    |      ●      |  ●  |     ●     |       ●       |   ●    |        ●         |        ●        |             |  ●   |
+| __0.2.0__        |   ●    |      ●      |  ●  |     ●     |       ●       |   ●    |        ●         |        ●        |             |  ●   |
+| __0.3.0 (next)__ |   ●    |      ●      |  ●  |     ●     |       ●       |   ●    |        ●         |        ○        |      ○      |  ●   |
\ No newline at end of file
diff --git a/docs/user-guide/sdks/overview.md b/docs/user-guide/sdks/overview.md
new file mode 100644
index 0000000000..f63af2a58a
--- /dev/null
+++ b/docs/user-guide/sdks/overview.md
@@ -0,0 +1,12 @@
+# Numaflow SDKs
+
+Numaflow is language-agnostic, letting you develop event-driven applications, stream processing, and serving pipelines in the language of your choice.
+
+From source to transformer to sink — each component in your pipeline can be written in any supported language, so your developers can focus on what they’re comfortable with.
+
+Available SDKs:
+
+- [**Java SDK**](https://github.com/numaproj/numaflow-java)  
+- [**Go SDK**](https://github.com/numaproj/numaflow-go)  
+- [**Python SDK**](https://github.com/numaproj/numaflow-python)  
+- [**Rust SDK**](https://github.com/numaproj/numaflow-rs)
diff --git a/mkdocs.yml b/mkdocs.yml
index daf0569a7b..a065b2f98d 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -40,10 +40,11 @@ nav:
       - quick-start.md
   - User Guide:
       - Core Concepts:
+          - Overview: "core-concepts/overview.md"
           - core-concepts/pipeline.md
-          - core-concepts/vertex.md
           - core-concepts/monovertex.md
           - core-concepts/serving.md
+          - core-concepts/vertex.md
           - core-concepts/inter-step-buffer.md
           - core-concepts/inter-step-buffer-service.md
           - core-concepts/watermarks.md
@@ -87,6 +88,10 @@ nav:
                   - Sliding: "user-guide/user-defined-functions/reduce/windowing/sliding.md"
                   - Session: "user-guide/user-defined-functions/reduce/windowing/session.md"
               - Examples: "user-guide/user-defined-functions/reduce/examples.md"
+      - SDKs:
+          - Overview: user-guide/sdks/overview.md
+          - Compatibility: user-guide/sdks/compatibility.md
+          - Features: user-guide/sdks/features.md
       - Reference:
           - user-guide/reference/pipeline-tuning.md
           - user-guide/reference/autoscaling.md


Reviews:
user: login: syayi
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-16T02:31:51Z
user: login: th0ger
body: Great work.
Added some minor typesetting edits.
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T07:20:57Z
user: login: BulkBeing
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T14:48:40Z

Review comments:
diff_hunk: @@ -0,0 +1,50 @@
+# Numaflow Core Concepts Overview
+
+Numaflow is a Kubernetes-native platform for event processing at scale, purpose-built for creating event-driven 
+applications, real-time stream processing pipelines, and serving systems. It provides a set of modular and composable 
+building blocks to help developers process events and serve efficiently in cloud-native environments.
+
+The following sections introduce the core concepts of Numaflow.
+
+* [MonoVertex](#monovertex)
+* [Pipeline](#pipeline)
+* [Serving](#serving)
+
+## MonoVertex
+
+[MonoVertex](./monovertex.md) is a lightweight way to develop event-driven applications in Numaflow. Each MonoVertex contains source, 
+transformer, and sink within a single unit, making it ideal for simple event processing tasks. The entire unit scales as
+one, making it a lightweight option for handling simple event processing patterns.
+
+### Use Cases
+
+- Stateless event transformation  
+- Filtering, mapping, or enriching event data  
+- Microservice-style event handling (e.g., call external APIs per event)  
+- Ingesting from or writing to external systems within a single unit  
+
+
+## Pipeline
+
+[Pipeline](pipeline.md) is designed for developing real-time stream processing pipelines. It allows you to connect multiple vertices—each
+representing a processing step—to handle data. Pipelines can include multiple sources and sinks, enabling integration with
+diverse systems at both the input and output stages.
+
+Each step in a pipeline can perform operations like transformation, routing, reduction, or aggregation, and is independently
+scalable. This makes Pipelines ideal for building robust, modular stream processing applications.
+
+### Use Cases
+- Aggregation and windowing (e.g., sum, count, reduce over time windows)  
+- Routing events based on payload or metadata  
+- Multi-step data processing and enrichment  
+- Scalable data pipelines for analytics and monitoring  
+
+## Serving
+
+[Serving](./serving.md) enables request/response interaction with streaming systems, allowing external clients to send data and 
+receive processed results in real time through interfaces like REST or Server-Sent Events (SSE)
path: docs/core-concepts/overview.md
user: login: th0ger
body: ```suggestion
receive processed results in real time through interfaces like REST or Server-Sent Events (SSE).
```
created_at: 2025-04-16T07:06:48Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,35 @@
+# SDK Compatibility
+
+SDK and Numaflow compatibility version matrix. 
+
+## Python
+
+| Python \ Numaflow  | v1.0.0 | v1.1.0 | v1.2.0 | v1.3.0 | v1.4.0 | v1.5.0 (next) |
+|:-------------------|:------:|:------:|:------:|:------:|:------:|:-------------:|
+| __v0.8.0__         |   ●    |   ●    |   ●    |   ●    |        |               |
+| __v0.9.0__         |        |        |        |        |   ●    |       ○       |
+| __v0.10.0 (next)__ |        |        |        |        |        |       ○       |
+
+## Golang
+
+| Golang  \ Numaflow | v1.0.0 | v1.1.0 | v1.2.0 | v1.3.0 | v1.4.0 | v1.5.0 (next) |
+|:-------------------|:------:|:------:|:------:|:------:|:------:|:-------------:|
+| __v0.8.0__         |   ●    |   ●    |   ●    |   ●    |        |               |
+| __v0.9.0__         |        |        |        |        |   ●    |       ○       |
+| __v0.10.0 (next)__ |        |        |        |        |        |       ○       |
+
+## Java
+
+| Java \ Numaflow    | v1.0.0 | v1.1.0 | v1.2.0 | v1.3.0 | v1.4.0 | v1.5.0 (next) |
+|:-------------------|:------:|:------:|:------:|:------:|:------:|:-------------:|
+| __v0.8.0__         |   ●    |   ●    |   ●    |   ●    |        |               |
+| __v0.9.0__         |        |        |        |        |   ●    |       ○       |
+| __v0.10.0 (next)__ |        |        |        |        |        |       ○       |
+
+## Rust
+
+| Rust \ Numaflow Core     | v1.0.0 | v1.1.0 | v1.2.0 | v1.3.0 | v1.4.0 | v1.5.0 (next) |
path: docs/user-guide/sdks/compatibility.md
user: login: th0ger
body: ```suggestion
| Rust \ Numaflow          | v1.0.0 | v1.1.0 | v1.2.0 | v1.3.0 | v1.4.0 | v1.5.0 (next) |
```
created_at: 2025-04-16T07:07:23Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,12 @@
+# Numaflow SDKs
+
+Numaflow is language-agnostic, that lets you develop event-driven applications, stream processing, and serving pipelines in the language of your choice.
path: docs/user-guide/sdks/overview.md
user: login: th0ger
body: ```suggestion
Numaflow is language-agnostic, letting you develop event-driven applications, stream processing, and serving pipelines in the language of your choice.
```
created_at: 2025-04-16T07:08:34Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,50 @@
+# Numaflow Core Concepts Overview
+
+Numaflow is a Kubernetes-native platform for event processing at scale, purpose-built for creating event-driven 
+applications, real-time stream processing pipelines, and serving systems. It provides a set of modular and composable 
+building blocks to help developers process events and serve efficiently in cloud-native environments.
+
+The following sections introduce the core concepts of Numaflow.
+
+* [MonoVertex](#monovertex)
+* [Pipeline](#pipeline)
+* [Serving](#serving)
+
+## MonoVertex
+
+[MonoVertex](./monovertex.md) is a lightweight way to develop event-driven applications in Numaflow. Each MonoVertex contains source, 
+transformer, and sink within a single unit, making it ideal for simple event processing tasks. The entire unit scales as
+one, making it a lightweight option for handling simple event processing patterns.
+
+### Use Cases
+
+- Stateless event transformation  
+- Filtering, mapping, or enriching event data  
+- Microservice-style event handling (e.g., call external APIs per event)  
+- Ingesting from or writing to external systems within a single unit  
+
+
+## Pipeline
+
+[Pipeline](pipeline.md) is designed for developing real-time stream processing pipelines. It allows you to connect multiple vertices—each
+representing a processing step—to handle data. Pipelines can include multiple sources and sinks, enabling integration with
path: docs/core-concepts/overview.md
user: login: th0ger
body: ```suggestion
[Pipeline](pipeline.md) is designed for developing real-time stream processing pipelines. It allows you to connect multiple vertices — each
representing a processing step — to handle data. Pipelines can include multiple sources and sinks, enabling integration with
```
I find the spaced em-dashes more readable, but it's a matter of style convention.

created_at: 2025-04-16T07:19:51Z
author_association: CONTRIBUTOR

PR comments:
user: login: codecov[bot]
created_at: 2025-04-14T18:24:18Z
body: ## [Codecov](https://app.codecov.io/gh/numaproj/numaflow/pull/2552?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=numaproj) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 69.53%. Comparing base [(`eb4dd43`)](https://app.codecov.io/gh/numaproj/numaflow/commit/eb4dd43995f96739f725d86fc80c7d890b21f19e?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=numaproj) to head [(`13d0f5c`)](https://app.codecov.io/gh/numaproj/numaflow/commit/13d0f5c9d79ae88246aa16b7bec12189f226a1f1?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=numaproj).
> Report is 1 commits behind head on main.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main    #2552      +/-   ##
==========================================
- Coverage   69.59%   69.53%   -0.07%     
==========================================
  Files         391      391              
  Lines       58130    58130              
==========================================
- Hits        40458    40418      -40     
- Misses      16589    16638      +49     
+ Partials     1083     1074       -9     
```

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/numaproj/numaflow/pull/2552?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=numaproj).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=numaproj).

<details><summary> :rocket: New features to boost your workflow: </summary>

- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
author_association: NONE
user: login: vigith
created_at: 2025-04-16T01:39:23Z
body: @th0ger please review
author_association: MEMBER
user: login: vigith
created_at: 2025-04-16T14:19:16Z
body: > Great work. Added some minor typesetting edits.

I have accepted the comments. Thank you :)
author_association: MEMBER