Project: Numaflow (App Definition and Development / Streaming & Messaging)
File: overview.md

Source vertex is responsible for reliable reading data from an unbounded source into Numaflow. Source vertex may require or formatting of data prior to sending it to the output buffers. Source Vertex also does tracking and late data detection. In Numaflow, we currently support the following sources A user-defined source is a custom source that a user can write using Numaflow SDK when the user needs to read data from a system that is not supported by the platform's built-in sources. User-defined source also supports custom acknowledge management for exactly-once reading. n container (platform code) and the sidecar container (user code) is through gRPC over Unix Domain Socket. There are some that can be used directly. You can build your own transformer in multiple languages. A user-defined transformer could be as simple as the example below in Golang. In the example, the transformer extracts event times from `timestamp` of the JSON payload and assigns them to messages as new event times. It also filters out unwanted messages based on `filterOut` of the payload. ```golang package main import ( "context" "encoding/json" "log" "time" "github.com/numaproj/numaflow-go/pkg/sourcetransformer" ) func transform(_ context.Context, keys []string, data sourcetransformer.Datum) sourcetransformer.Messages { /* Input messages are in JSON format. Sample: {"timestamp": "1673239888", "filterOut": "true"}. Field "timestamp" shows the real event time of the message, in the format of epoch. Field "filterOut" indicates whether the message should be filtered out, in the format of boolean. */ var jsonObject map[string]interface{} json.Unmarshal(data.Value(), &jsonObject) // event time assignment eventTime := data.EventTime() // if timestamp field exists, extract event time from payload. if ts, ok := jsonObject["timestamp"]; ok { eventTime = time.Unix(int64(ts.(float64)), 0) } // data filtering var filterOut bool if f, ok := jsonObject["filterOut"]; ok { filterOut = f.(bool) } if filterOut { return sourcetransformer.MessagesBuilder().Append(sourcetransformer.MessageToDrop(eventTime)) } else { return sourcetransformer.MessagesBuilder().Append(sourcetransformer.NewMessage(data.Value(), eventTime).WithKeys(keys)) } } func main() { err := sourcetransformer.NewServer(sourcetransformer.SourceTransformFunc(transform)).Start(context.Background()) if err != nil { log.Panic("Failed to start source transform server: ", err) } } ``` Check the links below to see another transformer example in various programming languages, where we apply conditional forwarding based on the input event time. After building a docker image for the written transformer, specify the image as below in the source vertex spec. ```yaml spec: vertices: name: my-vertex source: http: {} transformer: container: image: my-python-transformer-example:latest ``` Some environment variables are available in the source transformer container, they might be useful in you own source data transformer implementation. `NUMAFLOW_NAMESPACE` - Namespace. `NUMAFLOW_POD` - Pod name. `NUMAFLOW_REPLICA` - Replica index. `NUMAFLOWPIPELINENAME` - Name of the pipeline. `NUMAFLOWVERTEXNAME` - Name of the vertex. Configuration data can be provided to the transformer container at runtime multiple ways. `args` `command`