Repository: redpanda-data/rp-connect-docs
Pull request #186: DOC-514 Add mongodb_cdc_connector
Author: login: asimms41
Created: 2025-03-06T17:55:26Z | Merged: 2025-03-11T15:48:50Z

## Description

Resolves [DOC-514](https://redpandadata.atlassian.net/browse/DOC-514)
Review deadline: 7th March

Related PR to enable connector in cloud-docs: https://github.com/redpanda-data/cloud-docs/pull/227

This pull request introduces a new MongoDB CDC (Change Data Capture) input component to the documentation. The most important changes include updating the navigation to include the new component and adding comprehensive documentation for configuring and using the `mongodb_cdc` input.

Updates to documentation:

* [`modules/ROOT/nav.adoc`](diffhunk://#diff-5daf5a85fd51578ad9fc545388bf62176bcea32094ee97b5e30c2eea044e4193R67): Added a reference to the new MongoDB CDC input component in the navigation.
* [`modules/components/pages/inputs/mongodb_cdc.adoc`](diffhunk://#diff-826031304e0d00bc21e9c843f7f631ff2a5d78a926ec751e5ab4a605dabc104bR1-R393): Added detailed documentation for the `mongodb_cdc` input, including configuration fields, prerequisites, data capture methods, data replication modes, data mappings, metadata, and field descriptions.

## Page previews

[mongo_cdc connector](https://deploy-preview-186--redpanda-connect.netlify.app/redpanda-connect/components/inputs/mongodb_cdc/)

## Checks

- [x] New feature
- [ ] Content gap
- [ ] Support Follow-up
- [ ] Small fix (typos, links, copyedits, etc)

[DOC-514]: https://redpandadata.atlassian.net/browse/DOC-514?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ


Diff:
diff --git a/modules/ROOT/nav.adoc b/modules/ROOT/nav.adoc
index 7a69ea66..fb67a1e5 100644
--- a/modules/ROOT/nav.adoc
+++ b/modules/ROOT/nav.adoc
@@ -64,6 +64,7 @@
 *** xref:components:inputs/kafka.adoc[]
 *** xref:components:inputs/kafka_franz.adoc[]
 *** xref:components:inputs/mongodb.adoc[]
+*** xref:components:inputs/mongodb_cdc.adoc[]
 *** xref:components:inputs/mqtt.adoc[]
 *** xref:components:inputs/mysql_cdc.adoc[]
 *** xref:components:inputs/nanomsg.adoc[]
diff --git a/modules/components/pages/inputs/mongodb_cdc.adoc b/modules/components/pages/inputs/mongodb_cdc.adoc
new file mode 100644
index 00000000..53c64890
--- /dev/null
+++ b/modules/components/pages/inputs/mongodb_cdc.adoc
@@ -0,0 +1,392 @@
+= mongodb_cdc
+:page-beta: true
+// tag::single-source[]
+:type: input
+:categories: ["Services"]
+
+component_type_dropdown::[]
+
+Streams data changes from a MongoDB replica set, using MongoDB's https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data updates.
+
+ifndef::env-cloud[]
+Introduced in version 4.48.1.
+endif::[]
+
+[tabs]
+======
+Common::
++
+--
+
+```yml
+# Configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: "" # Optional
+    password: "" # Optional
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    auto_replay_nacks: true
+```
+
+--
+Advanced::
++
+--
+
+```yml
+# All configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: ""
+    password: ""
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    snapshot_auto_bucket_sharding: false
+    document_mode: update_lookup
+    json_marshal_mode: canonical
+    app_name: benthos
+    auto_replay_nacks: true
+```
+--
+======
+
+== Prerequisites
+
+- MongoDB version 6 or later
+ifndef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment.
+endif::[]
+ifdef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment. For detailed networking information, including how to set up a VPC peering connection, see xref:networking:index.adoc[Redpanda Cloud Networking].
+endif::[]
+- A MongoDB database running as a https://www.mongodb.com/docs/manual/replication/#replication-in-mongodb[replica set^] or in a https://www.mongodb.com/docs/manual/sharding/[sharded cluster^] using replica set https://www.mongodb.com/docs/manual/reference/replica-configuration/#rsconf.protocolVersion[protocol version 1].
+- A MongoDB database using the https://www.mongodb.com/docs/manual/core/wiredtiger/#storage-wiredtiger[WiredTiger] storage engine.
+
+
+ifndef::env-cloud[]
+== Enable connectivity from cloud-based data sources
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the IP addresses of your Redpanda Connect instances to your firewall rules.
+
+endif::[]
+
+ifdef::env-cloud[]
+
+== Enable connectivity from cloud-based data sources (BYOC)
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the NAT Gateway IP address of your Redpanda cluster to the allowlist of your data source.
+
+endif::[]
+
+== Data capture method
+
+The `mongodb_cdc` input uses https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data changes, which does not propagate _all_ changes to Redpanda Connect. To capture all changes in a MongoDB cluster, including deletions, enable pre- and post-image saving for the cluster and <<collections,required collections>>. For more information, see <<document_mode,`document_mode` options>> and the https://www.mongodb.com/docs/manual/changeStreams/#change-streams-with-document-pre--and-post-images[MongoDB documentation].
+
+
+== Data replication
+
+Redpanda Connect allows you to specify which <<collections,database collections>> in your source database to receive changes from.
+
+You can also run the `mongodb_cdc` input in one of two modes, depending on whether you need a snapshot of existing data before streaming updates.
+
+- Snapshot mode: Redpanda Connect first captures a snapshot of all data in the selected collections and streams the contents before processing changes from the last recorded https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^] position.
+- Streaming mode: Redpanda Connect skips the snapshot and processes only the most recent data changes, starting from the latest oplog position.
+
+=== Snapshot mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `true`, Redpanda Connect connects to your MongoDB database and does the following to capture a snapshot of all data in the selected collections:
+
+. Records the latest oplog position.
+. Determines the strategy for splitting the snapshot data down into shards or chunks for more efficient processing:
+.. If <<snapshot_auto_bucket_sharding,`snapshot_auto_bucket_sharding`>> is set to `false`, the internal `$splitVector` command is used to compute shards.  
+.. If <<snapshot_auto_bucket_sharding,`snapshot_auto_bucket_sharding`>> is set to `true`, the https://www.mongodb.com/docs/manual/reference/operator/aggregation/bucketAuto/[`$bucketAuto`^] command is used instead. This setting is for environments, such as MongoDB Atlas, where the `$splitVector` command is not available.
+. This input then uses the number of connections specified in <<snapshot_parallelism,`snapshot-parallelism`>> to read the selected collections.
++
+NOTE: If the pipeline restarts during this process, Redpanda Connect must start the snapshot capture from scratch to store the current oplog position in the <<checkpoint_cache,`checkpoint_cache`>>.
+. Finally, the input uses the stored oplog position to catch up with changes that occurred during snapshot processing.
+
+=== Streaming mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `false`, Redpanda Connect connects to your MongoDB database and starts processing data changes from the latest oplog position. If the pipeline restarts, Redpanda Connect resumes processing updates from the last oplog position written to the <<checkpoint_cache,`checkpoint_cache`>>.
+
+== Metadata
+
+This input adds the following metadata fields to each message:
+
+- `operation`: The type of data change that generated the message: `read`, `create`, `update`, `replace`, `delete`, `update`. A `read` operation occurs when the initial snapshot of the database is processed.
+- `collection`: The name of the collection from which the message originated.
+- `operation_time`: The time the data change was written to the https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^] in the form of a Binary JSON (BSON) timestamp: `{"t": <unix seconds>, "i": <sequence number>}`.
+
+== Fields
+
+=== `url`
+
+The URL of the target MongoDB server.
+
+*Type*: `string`
+
+*Default*: `""`
+
+```yml
+# Examples
+
+url: mongodb://localhost:27017
+```
+
+=== `database`
+
+The name of the MongoDB database to stream changes from.
+
+*Type*: `string`
+
+*Default*: `""`
+
+
+=== `username`
+
+The username to connect to the database.
+
+*Type*: `string`
+
+*Default*: `""`
+
+=== `password`
+
+The password to connect to the database.
+
+include::components:partial$secret_warning.adoc[]
+
+*Type*: `string`
+
+*Default*: `""`
+
+
+=== `collections`
+
+A list of collections to stream changes from. Specify each collection name as a separate item.
+
+*Type*: `array`
+
+*Default*: `[]`
+
+```yml
+# Examples
+
+collections:
+  - orders
+  - customers
+  - inventory
+```
+
+=== `checkpoint_cache`
+
+Specify a xref:components:caches/about.adoc[`cache` resource] to store the oplog position for the most recent data update streamed to Redpanda Connect. After a restart, Redpanda Connect can continue processing changes from this position, avoiding the need to reprocess all collection updates.
+
+*Type*: `string`
+
+*Default*: `""`
+
+```yml
+# Examples
+
+input:
+  mongodb_cdc:
+    url: mongodb://localhost:27017
+    collections: [my_collection]
+    checkpoint_cache: "my_cdc_cache"
+cache_resources:
+  - label: "my_cdc_cache"
+    redis:
+      url: redis://:6379
+```
+
+=== `checkpoint_key`
+
+The key identifier used to store the oplog position in <<checkpoint_cache,`checkpoint_cache`>>. If you have multiple `mongodb_cdc` inputs sharing the same cache, you can provide an alternative key.
+
+
+*Type*: `string`
+
+*Default*: `mongodb_cdc_checkpoint`
+
+
+=== `checkpoint_interval`
+
+The interval between writing checkpoints to the cache.
+
+*Type*: `string`
+
+*Default*: `5s`
+
+=== `checkpoint_limit`
+
+The maximum number of in-flight messages emitted from this input. Increasing this limit enables parallel processing, and batching at the output level. To preserve at-least-once guarantees, any given oplog position is not acknowledged until all messages under that offset are delivered.
+
+*Type*: `int`
+
+*Default*: `1000`
+
+
+=== `read_batch_size`
+
+The number of documents to fetch in each message batch from MongoDB.
+
+
+*Type*: `int`
+
+*Default*: `1000`
+
+
+=== `read_max_wait`
+
+The maximum duration MongoDB waits to accumulate the <<read_batch_size,`read_batch_size`>> documents on a change stream before returning the batch to Redpanda Connect.
+
+*Type*: `string`
+
+*Default*: `1s`
+
+=== `stream_snapshot`
+
+When set to `true`, this input streams a snapshot of all existing data in the source collections before streaming data changes.
+
+*Type*: `bool`
+
+*Default*: `false`
+
+```yml
+# Examples
+
+stream_snapshot: true
+```
+
+=== `snapshot_parallelism`
+
+Specifies the number of connections to use when reading the initial snapshot from one or more collections. Increase this number to enable parallel processing of the snapshot. 
+
+This feature uses the `$splitVector` command to split snapshot data into chunks for more efficient processing.
+
+This field is only applicable when `stream_snapshot` is set to `true`. 
+
+*Type*: `int`
+
+*Default*: `1`
+
+=== `snapshot_auto_bucket_sharding`
+
+Uses the https://www.mongodb.com/docs/manual/reference/operator/aggregation/bucketAuto/[`$bucketAuto`^] command instead of the default, `$splitVector`, to split the snapshot data into chunks for processing. This is required for environments, such as MongoDB Atlas, where the `$splitVector` command is not available. To enable parallel processing in these environments:
+
+- Set this field to to `true`.
+- Set `stream_snapshot` to `true`.
+- Increase `snapshot_parallelism` to a value greater than `1`.
+
+*Type*: `bool`
+
+*Default*: `false`
+
+=== `document_mode`
+
+The mode in which MongoDB emits document changes to Redpanda Connect, specifically updates and deletes.
+
+*Type*: `string`
+
+*Default*: `update_lookup`
+
+[cols="1,2"]
+|===
+| Option | Description
+
+| `update_lookup`
+a| Emits full documents for insert, replace, and update operations, but only the `_id` field for delete operations.  
+This corresponds to the `updateLookup` option.  
+For more information, see the https://www.mongodb.com/docs/manual/changeStreams/#std-label-change-streams-updateLookup[MongoDB documentation^].
+
+| `pre_and_post_images`
+a| Emits full documents for update and delete operations by using pre- and post-image collections.  
+For setup and configuration details, see the https://www.mongodb.com/docs/manual/changeStreams/#change-streams-with-document-pre--and-post-images[MongoDB documentation^].
+
+
+| `partial_update`
+a| Update operations contain only a description of the changes made to the document, using the following schema:
+
+[,bash,role="no-placeholders"]
+----
+{
+  "_id": "<document_id>",
+  "operations": [
+    # type == set: Indicates that a value was updated. For example:
+    # root.name."address.city" = "New York"
+    {"path": ["name", "address.city"], "type": "set", "value": "New York"},
+
+    # type == unset: Indicates that a value was deleted. For example:
+    # root.phone = deleted()
+    {"path": ["phone"], "type": "unset", "value": null},
+
+    # type == truncatedArray: Indicates that an array was truncated to a specified number of elements. For example:
+    # root.items = this.items.slice(2)
+    {"path": ["items"], "type": "truncatedArray", "value": 2}
+  ]
+}
+----
+
+|===
+
+=== `json_marshal_mode`
+
+Controls the format used to convert a message from BSON to JSON when it is received by Redpanda Connect.
+
+*Type*: `string`
+
+*Default*: `canonical`
+
+|===
+| Option | Description
+
+| `canonical`
+| A string format that preserves the exact data types of values stored in MongoDB at the expense of readability and interoperability. Conversion from `canonical` to JSON generally preserves type information.
+| `relaxed`
+| A string format that is human-readable and compatible with most JSON parsers but may lose type fidelity. Conversion from `relaxed` format to JSON can lose type information.
+
+|===
+
+=== `app_name`
+
+The client application name.
+
+*Type*: `string`
+
+*Default*: `benthos`
+
+
+=== `auto_replay_nacks`
+
+Whether to automatically replay rejected messages (negative acknowledgements) at the output level. If the cause of rejections is persistent, leaving this option enabled can result in back pressure.
+
+Set `auto_replay_nacks` to `false` to delete rejected messages. Disabling auto replays can greatly improve memory efficiency of high throughput streams as the original shape of the data is discarded immediately upon consumption and mutation.
+
+
+*Type*: `bool`
+
+*Default*: `true`
+
+// end::single-source[]


Reviews:
user: login: asimms41
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T17:56:30Z
user: login: asimms41
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T17:58:17Z
user: login: asimms41
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T18:13:26Z
user: login: asimms41
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T18:14:09Z
user: login: asimms41
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T18:14:37Z
user: login: asimms41
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T18:15:21Z
user: login: asimms41
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T18:17:21Z
user: login: asimms41
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T18:18:01Z
user: login: rockwotj
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T08:51:06Z
user: login: asimms41
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T09:12:35Z
user: login: asimms41
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T09:14:12Z
user: login: asimms41
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T09:16:21Z
user: login: asimms41
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T09:24:30Z
user: login: asimms41
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T09:28:15Z
user: login: rockwotj
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T09:29:36Z
user: login: rockwotj
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T09:31:42Z
user: login: rockwotj
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T09:33:02Z
user: login: asimms41
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T09:57:56Z
user: login: asimms41
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T10:45:42Z
user: login: asimms41
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T10:46:01Z
user: login: asimms41
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T10:46:18Z
user: login: asimms41
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T10:48:29Z
user: login: asimms41
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T10:49:06Z
user: login: rockwotj
body: LGTM
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T15:59:46Z
user: login: asimms41
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T16:35:11Z
user: login: asimms41
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T16:35:22Z
user: login: JakeSCahill
body: One suggested fix and a general question
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T16:38:11Z
user: login: asimms41
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T16:44:34Z
user: login: asimms41
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T16:46:28Z
user: login: JakeSCahill
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T16:56:54Z
user: login: jrkinley
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-11T09:28:42Z

Review comments:
diff_hunk: @@ -0,0 +1,393 @@
+= mongodb_cdc
+// tag::single-source[]
+:type: input
+:page-beta: true
+:categories: ["Services"]
+
+component_type_dropdown::[]
+
+Streams data changes from a MongoDB replica set, using MongoDB's https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data updates.
+
+ifndef::env-cloud[]
+Introduced in version 4.48.1.
+endif::[]
+
+[tabs]
+======
+Common::
++
+--
+
+```yml
+# Configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: "" # Optional
+    password: "" # Optional
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    auto_replay_nacks: true
+```
+
+--
+Advanced::
++
+--
+
+```yml
+# All configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: ""
+    password: ""
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    snapshot_auto_bucket_sharding: false
+    document_mode: update_lookup
+    json_marshal_mode: canonical
+    app_name: benthos
+    auto_replay_nacks: true
+```
+--
+======
+
+== Prerequisites
+
+- MongoDB version 6 or later
path: modules/components/pages/inputs/mongodb_cdc.adoc
user: login: asimms41
body: Please check for accuracy.
created_at: 2025-03-06T17:56:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,393 @@
+= mongodb_cdc
+// tag::single-source[]
+:type: input
+:page-beta: true
+:categories: ["Services"]
+
+component_type_dropdown::[]
+
+Streams data changes from a MongoDB replica set, using MongoDB's https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data updates.
+
+ifndef::env-cloud[]
+Introduced in version 4.48.1.
+endif::[]
+
+[tabs]
+======
+Common::
++
+--
+
+```yml
+# Configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: "" # Optional
+    password: "" # Optional
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    auto_replay_nacks: true
+```
+
+--
+Advanced::
++
+--
+
+```yml
+# All configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: ""
+    password: ""
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    snapshot_auto_bucket_sharding: false
+    document_mode: update_lookup
+    json_marshal_mode: canonical
+    app_name: benthos
+    auto_replay_nacks: true
+```
+--
+======
+
+== Prerequisites
+
+- MongoDB version 6 or later
+ifndef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment.
+endif::[]
+ifdef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment. For detailed networking information, including how to set up a VPC peering connection, see xref:networking:index.adoc[Redpanda Cloud Networking].
+endif::[]
+- A sharded MongoDB database or MongoDB cluster running as a replica set
+
+ifndef::env-cloud[]
+== Enable connectivity from cloud-based data sources
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the IP addresses of your Redpanda Connect instances to your firewall rules.
+
+endif::[]
+
+ifdef::env-cloud[]
+
+== Enable connectivity from cloud-based data sources (BYOC)
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the NAT Gateway IP address of your Redpanda cluster to the allowlist of your data source.
+
+endif::[]
+
+== Data capture method
+
+The `mongodb_cdc` input uses https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data changes, which does not propagate _all_ changes to Redpanda Connect. To capture all changes in a MongoDB cluster, including deletions, enable pre- and post-image saving for the cluster and <<database-collections, required collections>>. For more information, see <<document-mode, `document_mode` options>> and the https://www.mongodb.com/docs/manual/changeStreams/#change-streams-with-document-pre--and-post-images[MongoDB documentation].
+
+
+== Data replication
+
+Redpanda Connect allows you to specify which <<collections,database collections>> in your source database to receive changes from.
+
+You can also run the `mongodb_cdc` input in one of two modes, depending on whether you need a snapshot of existing data before streaming updates.
+
+- Snapshot mode: Redpanda Connect first captures a snapshot of all data in the selected collections and streams the contents before processing changes from the last recorded https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^] position.
+- Streaming mode: Redpanda Connect skips the snapshot and processes only the most recent data changes, starting from the latest oplog position.
+
+=== Snapshot mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `true`, Redpanda Connect connects to your MongoDB database and does the following to capture a snapshot of all data in the selected collections:
+
+//TODO should include parallel processing of snapshots and document_mode
path: modules/components/pages/inputs/mongodb_cdc.adoc
user: login: asimms41
body: @rockwotj do you have any notes on this process for taking snapshots as for https://docs.redpanda.com/redpanda-connect/components/inputs/mysql_cdc/
created_at: 2025-03-06T17:58:17Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,393 @@
+= mongodb_cdc
+// tag::single-source[]
+:type: input
+:page-beta: true
+:categories: ["Services"]
+
+component_type_dropdown::[]
+
+Streams data changes from a MongoDB replica set, using MongoDB's https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data updates.
+
+ifndef::env-cloud[]
+Introduced in version 4.48.1.
+endif::[]
+
+[tabs]
+======
+Common::
++
+--
+
+```yml
+# Configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: "" # Optional
+    password: "" # Optional
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    auto_replay_nacks: true
+```
+
+--
+Advanced::
++
+--
+
+```yml
+# All configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: ""
+    password: ""
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    snapshot_auto_bucket_sharding: false
+    document_mode: update_lookup
+    json_marshal_mode: canonical
+    app_name: benthos
+    auto_replay_nacks: true
+```
+--
+======
+
+== Prerequisites
+
+- MongoDB version 6 or later
+ifndef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment.
+endif::[]
+ifdef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment. For detailed networking information, including how to set up a VPC peering connection, see xref:networking:index.adoc[Redpanda Cloud Networking].
+endif::[]
+- A sharded MongoDB database or MongoDB cluster running as a replica set
+
+ifndef::env-cloud[]
+== Enable connectivity from cloud-based data sources
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the IP addresses of your Redpanda Connect instances to your firewall rules.
+
+endif::[]
+
+ifdef::env-cloud[]
+
+== Enable connectivity from cloud-based data sources (BYOC)
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the NAT Gateway IP address of your Redpanda cluster to the allowlist of your data source.
+
+endif::[]
+
+== Data capture method
+
+The `mongodb_cdc` input uses https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data changes, which does not propagate _all_ changes to Redpanda Connect. To capture all changes in a MongoDB cluster, including deletions, enable pre- and post-image saving for the cluster and <<database-collections, required collections>>. For more information, see <<document-mode, `document_mode` options>> and the https://www.mongodb.com/docs/manual/changeStreams/#change-streams-with-document-pre--and-post-images[MongoDB documentation].
+
+
+== Data replication
+
+Redpanda Connect allows you to specify which <<collections,database collections>> in your source database to receive changes from.
+
+You can also run the `mongodb_cdc` input in one of two modes, depending on whether you need a snapshot of existing data before streaming updates.
+
+- Snapshot mode: Redpanda Connect first captures a snapshot of all data in the selected collections and streams the contents before processing changes from the last recorded https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^] position.
+- Streaming mode: Redpanda Connect skips the snapshot and processes only the most recent data changes, starting from the latest oplog position.
+
+=== Snapshot mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `true`, Redpanda Connect connects to your MongoDB database and does the following to capture a snapshot of all data in the selected collections:
+
+//TODO should include parallel processing of snapshots and document_mode
+
+NOTE: If the pipeline restarts during this process, Redpanda Connect must start the snapshot capture from scratch to store the current oplog position in the <<checkpoint_cache,`checkpoint_cache`>>.
+
+//TODO
path: modules/components/pages/inputs/mongodb_cdc.adoc
user: login: asimms41
body: Just a placeholder in case the process for taking a snapshot is split into two stages.
created_at: 2025-03-06T18:13:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,393 @@
+= mongodb_cdc
+// tag::single-source[]
+:type: input
+:page-beta: true
+:categories: ["Services"]
+
+component_type_dropdown::[]
+
+Streams data changes from a MongoDB replica set, using MongoDB's https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data updates.
+
+ifndef::env-cloud[]
+Introduced in version 4.48.1.
+endif::[]
+
+[tabs]
+======
+Common::
++
+--
+
+```yml
+# Configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: "" # Optional
+    password: "" # Optional
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    auto_replay_nacks: true
+```
+
+--
+Advanced::
++
+--
+
+```yml
+# All configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: ""
+    password: ""
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    snapshot_auto_bucket_sharding: false
+    document_mode: update_lookup
+    json_marshal_mode: canonical
+    app_name: benthos
+    auto_replay_nacks: true
+```
+--
+======
+
+== Prerequisites
+
+- MongoDB version 6 or later
+ifndef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment.
+endif::[]
+ifdef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment. For detailed networking information, including how to set up a VPC peering connection, see xref:networking:index.adoc[Redpanda Cloud Networking].
+endif::[]
+- A sharded MongoDB database or MongoDB cluster running as a replica set
+
+ifndef::env-cloud[]
+== Enable connectivity from cloud-based data sources
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the IP addresses of your Redpanda Connect instances to your firewall rules.
+
+endif::[]
+
+ifdef::env-cloud[]
+
+== Enable connectivity from cloud-based data sources (BYOC)
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the NAT Gateway IP address of your Redpanda cluster to the allowlist of your data source.
+
+endif::[]
+
+== Data capture method
+
+The `mongodb_cdc` input uses https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data changes, which does not propagate _all_ changes to Redpanda Connect. To capture all changes in a MongoDB cluster, including deletions, enable pre- and post-image saving for the cluster and <<database-collections, required collections>>. For more information, see <<document-mode, `document_mode` options>> and the https://www.mongodb.com/docs/manual/changeStreams/#change-streams-with-document-pre--and-post-images[MongoDB documentation].
+
+
+== Data replication
+
+Redpanda Connect allows you to specify which <<collections,database collections>> in your source database to receive changes from.
+
+You can also run the `mongodb_cdc` input in one of two modes, depending on whether you need a snapshot of existing data before streaming updates.
+
+- Snapshot mode: Redpanda Connect first captures a snapshot of all data in the selected collections and streams the contents before processing changes from the last recorded https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^] position.
+- Streaming mode: Redpanda Connect skips the snapshot and processes only the most recent data changes, starting from the latest oplog position.
+
+=== Snapshot mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `true`, Redpanda Connect connects to your MongoDB database and does the following to capture a snapshot of all data in the selected collections:
+
+//TODO should include parallel processing of snapshots and document_mode
+
+NOTE: If the pipeline restarts during this process, Redpanda Connect must start the snapshot capture from scratch to store the current oplog position in the <<checkpoint_cache,`checkpoint_cache`>>.
+
+//TODO
+
+Finally, the input uses the stored oplog position to catch up with changes that occurred during snapshot processing.
+
+=== Streaming mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `false`, Redpanda Connect connects to your MongoDB database and starts processing data changes from the latest oplog position. If the pipeline restarts, Redpanda Connect resumes processing updates from the last oplog position written to the <<checkpoint_cache,`checkpoint_cache`>>.
+
+== Data mappings
path: modules/components/pages/inputs/mongodb_cdc.adoc
user: login: asimms41
body: I don't think this section is required for this connector, but please confirm.
created_at: 2025-03-06T18:14:09Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,393 @@
+= mongodb_cdc
+// tag::single-source[]
+:type: input
+:page-beta: true
+:categories: ["Services"]
+
+component_type_dropdown::[]
+
+Streams data changes from a MongoDB replica set, using MongoDB's https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data updates.
+
+ifndef::env-cloud[]
+Introduced in version 4.48.1.
+endif::[]
+
+[tabs]
+======
+Common::
++
+--
+
+```yml
+# Configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: "" # Optional
+    password: "" # Optional
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    auto_replay_nacks: true
+```
+
+--
+Advanced::
++
+--
+
+```yml
+# All configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: ""
+    password: ""
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    snapshot_auto_bucket_sharding: false
+    document_mode: update_lookup
+    json_marshal_mode: canonical
+    app_name: benthos
+    auto_replay_nacks: true
+```
+--
+======
+
+== Prerequisites
+
+- MongoDB version 6 or later
+ifndef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment.
+endif::[]
+ifdef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment. For detailed networking information, including how to set up a VPC peering connection, see xref:networking:index.adoc[Redpanda Cloud Networking].
+endif::[]
+- A sharded MongoDB database or MongoDB cluster running as a replica set
+
+ifndef::env-cloud[]
+== Enable connectivity from cloud-based data sources
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the IP addresses of your Redpanda Connect instances to your firewall rules.
+
+endif::[]
+
+ifdef::env-cloud[]
+
+== Enable connectivity from cloud-based data sources (BYOC)
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the NAT Gateway IP address of your Redpanda cluster to the allowlist of your data source.
+
+endif::[]
+
+== Data capture method
+
+The `mongodb_cdc` input uses https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data changes, which does not propagate _all_ changes to Redpanda Connect. To capture all changes in a MongoDB cluster, including deletions, enable pre- and post-image saving for the cluster and <<database-collections, required collections>>. For more information, see <<document-mode, `document_mode` options>> and the https://www.mongodb.com/docs/manual/changeStreams/#change-streams-with-document-pre--and-post-images[MongoDB documentation].
+
+
+== Data replication
+
+Redpanda Connect allows you to specify which <<collections,database collections>> in your source database to receive changes from.
+
+You can also run the `mongodb_cdc` input in one of two modes, depending on whether you need a snapshot of existing data before streaming updates.
+
+- Snapshot mode: Redpanda Connect first captures a snapshot of all data in the selected collections and streams the contents before processing changes from the last recorded https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^] position.
+- Streaming mode: Redpanda Connect skips the snapshot and processes only the most recent data changes, starting from the latest oplog position.
+
+=== Snapshot mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `true`, Redpanda Connect connects to your MongoDB database and does the following to capture a snapshot of all data in the selected collections:
+
+//TODO should include parallel processing of snapshots and document_mode
+
+NOTE: If the pipeline restarts during this process, Redpanda Connect must start the snapshot capture from scratch to store the current oplog position in the <<checkpoint_cache,`checkpoint_cache`>>.
+
+//TODO
+
+Finally, the input uses the stored oplog position to catch up with changes that occurred during snapshot processing.
+
+=== Streaming mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `false`, Redpanda Connect connects to your MongoDB database and starts processing data changes from the latest oplog position. If the pipeline restarts, Redpanda Connect resumes processing updates from the last oplog position written to the <<checkpoint_cache,`checkpoint_cache`>>.
+
+== Data mappings
+
+The following table shows how selected MongoDB data types are mapped to data types supported in Redpanda Connect. All other data types are mapped to string values.
+
+|===
+| MongoDB data type | Bloblang value
+
+| Placeholder
+| Placeholder
+
+|===
+
+== Metadata
+
+This input adds the following metadata fields to each message:
+
+- `operation`: The type of data change that generated the message: `read`, `create`, `update`, `replace`, `delete`, `update`. A `read` operation occurs when the initial snapshot of the database is processed.
+- `collection`: The name of the collection from which the message originated.
+- `operation_time`: The time the data change was written to the https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^].
+
+== Fields
+
+=== `url`
+
+The URL of the target MongoDB server.
+
+*Type*: `string`
+
+*Default*: `""`
+
+```yml
+# Examples
+
+url: mongodb://localhost:27017
+```
+
+=== `database`
+
+The name of the MongoDB database to stream changes from.
+
+*Type*: `string`
+
+*Default*: `""`
+
+
+=== `username`
+
+The username to connect to the database.
+
+*Type*: `string`
+
+*Default*: `""`
+
+=== `password`
+
+The password to connect to the database.
+
+include::components:partial$secret_warning.adoc[]
+
+*Type*: `string`
+
+*Default*: `""`
+
+
+=== `collections`
+
+A list of collections to stream changes from. Specify each collection name as a separate item.
+
+*Type*: `array`
+
+*Default*: `[]`
+
+```yml
+# Examples
+
+collections:
+  - orders
+  - customers
+  - inventory
+```
+
+=== `checkpoint_cache`
+
+Specify a `cache` resource to store the oplog position for the most recent data update streamed to Redpanda Connect. After a restart, Redpanda Connect can continue processing changes from this position, avoiding the need to reprocess all collection updates.
+
+*Type*: `string`
+
+*Default*: `""`
+
+```yml
+# Examples
+
+input:
+  mongodb_cdc:
+    url: mongodb://localhost:27017
+    collections: [my_collection]
+    checkpoint_cache: "my_cdc_cache"
+cache_resources:
+  - label: "my_cdc_cache"
+    redis:
+      url: redis://:6379
+```
+
+=== `checkpoint_key`
+
+The key identifier used to store the oplog position in <<checkpoint_cache,`checkpoint_cache`>>. If you have multiple `mongodb_cdc` inputs sharing the same cache, you can provide an alternative key.
+
+
+*Type*: `string`
+
+*Default*: `mongodb_cdc_checkpoint`
+
+
+=== `checkpoint_interval`
+
+The interval between writing checkpoints to the cache.
+
+*Type*: `string`
+
+*Default*: `5s`
+
+=== `checkpoint_limit`
+
+The maximum number of messages that this input can process at a given time. Increasing this limit enables parallel processing, and batching at the output level. To preserve at-least-once guarantees, any given oplog position is not acknowledged until all messages under that offset are delivered.
path: modules/components/pages/inputs/mongodb_cdc.adoc
user: login: asimms41
body: Please check for accuracy.
created_at: 2025-03-06T18:14:37Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,393 @@
+= mongodb_cdc
+// tag::single-source[]
+:type: input
+:page-beta: true
+:categories: ["Services"]
+
+component_type_dropdown::[]
+
+Streams data changes from a MongoDB replica set, using MongoDB's https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data updates.
+
+ifndef::env-cloud[]
+Introduced in version 4.48.1.
+endif::[]
+
+[tabs]
+======
+Common::
++
+--
+
+```yml
+# Configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: "" # Optional
+    password: "" # Optional
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    auto_replay_nacks: true
+```
+
+--
+Advanced::
++
+--
+
+```yml
+# All configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: ""
+    password: ""
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    snapshot_auto_bucket_sharding: false
+    document_mode: update_lookup
+    json_marshal_mode: canonical
+    app_name: benthos
+    auto_replay_nacks: true
+```
+--
+======
+
+== Prerequisites
+
+- MongoDB version 6 or later
+ifndef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment.
+endif::[]
+ifdef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment. For detailed networking information, including how to set up a VPC peering connection, see xref:networking:index.adoc[Redpanda Cloud Networking].
+endif::[]
+- A sharded MongoDB database or MongoDB cluster running as a replica set
+
+ifndef::env-cloud[]
+== Enable connectivity from cloud-based data sources
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the IP addresses of your Redpanda Connect instances to your firewall rules.
+
+endif::[]
+
+ifdef::env-cloud[]
+
+== Enable connectivity from cloud-based data sources (BYOC)
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the NAT Gateway IP address of your Redpanda cluster to the allowlist of your data source.
+
+endif::[]
+
+== Data capture method
+
+The `mongodb_cdc` input uses https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data changes, which does not propagate _all_ changes to Redpanda Connect. To capture all changes in a MongoDB cluster, including deletions, enable pre- and post-image saving for the cluster and <<database-collections, required collections>>. For more information, see <<document-mode, `document_mode` options>> and the https://www.mongodb.com/docs/manual/changeStreams/#change-streams-with-document-pre--and-post-images[MongoDB documentation].
+
+
+== Data replication
+
+Redpanda Connect allows you to specify which <<collections,database collections>> in your source database to receive changes from.
+
+You can also run the `mongodb_cdc` input in one of two modes, depending on whether you need a snapshot of existing data before streaming updates.
+
+- Snapshot mode: Redpanda Connect first captures a snapshot of all data in the selected collections and streams the contents before processing changes from the last recorded https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^] position.
+- Streaming mode: Redpanda Connect skips the snapshot and processes only the most recent data changes, starting from the latest oplog position.
+
+=== Snapshot mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `true`, Redpanda Connect connects to your MongoDB database and does the following to capture a snapshot of all data in the selected collections:
+
+//TODO should include parallel processing of snapshots and document_mode
+
+NOTE: If the pipeline restarts during this process, Redpanda Connect must start the snapshot capture from scratch to store the current oplog position in the <<checkpoint_cache,`checkpoint_cache`>>.
+
+//TODO
+
+Finally, the input uses the stored oplog position to catch up with changes that occurred during snapshot processing.
+
+=== Streaming mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `false`, Redpanda Connect connects to your MongoDB database and starts processing data changes from the latest oplog position. If the pipeline restarts, Redpanda Connect resumes processing updates from the last oplog position written to the <<checkpoint_cache,`checkpoint_cache`>>.
+
+== Data mappings
+
+The following table shows how selected MongoDB data types are mapped to data types supported in Redpanda Connect. All other data types are mapped to string values.
+
+|===
+| MongoDB data type | Bloblang value
+
+| Placeholder
+| Placeholder
+
+|===
+
+== Metadata
+
+This input adds the following metadata fields to each message:
+
+- `operation`: The type of data change that generated the message: `read`, `create`, `update`, `replace`, `delete`, `update`. A `read` operation occurs when the initial snapshot of the database is processed.
+- `collection`: The name of the collection from which the message originated.
+- `operation_time`: The time the data change was written to the https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^].
+
+== Fields
+
+=== `url`
+
+The URL of the target MongoDB server.
+
+*Type*: `string`
+
+*Default*: `""`
+
+```yml
+# Examples
+
+url: mongodb://localhost:27017
+```
+
+=== `database`
+
+The name of the MongoDB database to stream changes from.
+
+*Type*: `string`
+
+*Default*: `""`
+
+
+=== `username`
+
+The username to connect to the database.
+
+*Type*: `string`
+
+*Default*: `""`
+
+=== `password`
+
+The password to connect to the database.
+
+include::components:partial$secret_warning.adoc[]
+
+*Type*: `string`
+
+*Default*: `""`
+
+
+=== `collections`
+
+A list of collections to stream changes from. Specify each collection name as a separate item.
+
+*Type*: `array`
+
+*Default*: `[]`
+
+```yml
+# Examples
+
+collections:
+  - orders
+  - customers
+  - inventory
+```
+
+=== `checkpoint_cache`
+
+Specify a `cache` resource to store the oplog position for the most recent data update streamed to Redpanda Connect. After a restart, Redpanda Connect can continue processing changes from this position, avoiding the need to reprocess all collection updates.
+
+*Type*: `string`
+
+*Default*: `""`
+
+```yml
+# Examples
+
+input:
+  mongodb_cdc:
+    url: mongodb://localhost:27017
+    collections: [my_collection]
+    checkpoint_cache: "my_cdc_cache"
+cache_resources:
+  - label: "my_cdc_cache"
+    redis:
+      url: redis://:6379
+```
+
+=== `checkpoint_key`
+
+The key identifier used to store the oplog position in <<checkpoint_cache,`checkpoint_cache`>>. If you have multiple `mongodb_cdc` inputs sharing the same cache, you can provide an alternative key.
+
+
+*Type*: `string`
+
+*Default*: `mongodb_cdc_checkpoint`
+
+
+=== `checkpoint_interval`
+
+The interval between writing checkpoints to the cache.
+
+*Type*: `string`
+
+*Default*: `5s`
+
+=== `checkpoint_limit`
+
+The maximum number of messages that this input can process at a given time. Increasing this limit enables parallel processing, and batching at the output level. To preserve at-least-once guarantees, any given oplog position is not acknowledged until all messages under that offset are delivered.
+
+*Type*: `int`
+
+*Default*: `1000`
+
+
+=== `read_batch_size`
+
+The number of documents to fetch in each message batch from MongoDB.
+
+
+*Type*: `int`
+
+*Default*: `1000`
+
+
+=== `read_max_wait`
+
+The maximum duration MongoDB waits to accumulate the <<read-batch-size, `read_batch_size`>> documents on a change stream before returning the batch to Redpanda Connect.
+
+*Type*: `string`
+
+*Default*: `1s`
+
+=== `stream_snapshot`
+
+When set to `true`, this input streams a snapshot of all existing data in the source collections before streaming data changes.
+
+*Type*: `bool`
+
+*Default*: `false`
+
+```yml
+# Examples
+
+stream_snapshot: true
+```
+
+=== `snapshot_parallelism`
+
+Specifies the number of connections to use when reading the initial snapshot from one or more collections. Increase this number to enable parallel processing of the snapshot.
+
+This feature uses the `$splitVector` command to split snapshot data into chunks for more efficient processing.
+
+*Type*: `int`
+
+*Default*: `1`
+
+=== `snapshot_auto_bucket_sharding`
+
+Uses the https://www.mongodb.com/docs/manual/reference/operator/aggregation/bucketAuto/[`$bucketAuto`^] command instead of the default, `$splitVector`, to split the snapshot data into chunks for processing. This is required for environments, such as MongoDB Atlas, where privileged access is restricted. To enable parallel processing in these environments:
+
+- Set `snapshot_auto_bucket_sharding` to `true`.
+- Increase `snapshot_parallelism` to a value greater than `1`.
+
+*Type*: `bool`
+
+*Default*: `false`
+
+=== `document_mode`
+
+The mode in which MongoDB emits document changes to Redpanda Connect, specifically updates and deletes.
+
+[cols="1,2"]
+|===
+| Option | Mode description
+
+| `partial_update`
+a| Update operations contain only a description of the changes made to the document, using the following schema:
+
+[,bash,role="no-placeholders"]
+----
+{
+  "_id": "<document_id>",
+  "operations": [
+    # type == set: Indicates that a value was updated. For example:
+    # root.name."address.city" = "New York"
+    {"path": ["name", "address.city"], "type": "set", "value": "New York"},
+
+    # type == unset: Indicates that a value was deleted. For example:
+    # root.phone = deleted()
+    {"path": ["phone"], "type": "unset", "value": null},
+
+    # type == truncatedArray: Indicates that an array was truncated to a specified number of elements. For example:
+    # root.items = this.items.slice(2)
+    {"path": ["items"], "type": "truncatedArray", "value": 2}
+  ]
+}
+----
+
+| `pre_and_post_images`
+a| Emits full documents for update and delete operations by using pre- and post-image collections.  
+For setup and configuration details, see the https://www.mongodb.com/docs/manual/changeStreams/#change-streams-with-document-pre--and-post-images[MongoDB documentation^].
+
+| `update_lookup`
+a| Emits full documents for insert, replace, and update operations, but only the `_id` field for delete operations.  
+This corresponds to the `updateLookup` option.  
+For more information, see the https://www.mongodb.com/docs/manual/changeStreams/#std-label-change-streams-updateLookup[MongoDB documentation^].
+
+|===
+
+=== `json_marshal_mode`
+
+Controls how MongoDB formats messages sent to Redpanda Connect.
path: modules/components/pages/inputs/mongodb_cdc.adoc
user: login: asimms41
body: Is this correct?
created_at: 2025-03-06T18:15:21Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,393 @@
+= mongodb_cdc
+// tag::single-source[]
+:type: input
+:page-beta: true
+:categories: ["Services"]
+
+component_type_dropdown::[]
+
+Streams data changes from a MongoDB replica set, using MongoDB's https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data updates.
+
+ifndef::env-cloud[]
+Introduced in version 4.48.1.
+endif::[]
+
+[tabs]
+======
+Common::
++
+--
+
+```yml
+# Configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: "" # Optional
+    password: "" # Optional
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    auto_replay_nacks: true
+```
+
+--
+Advanced::
++
+--
+
+```yml
+# All configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: ""
+    password: ""
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    snapshot_auto_bucket_sharding: false
+    document_mode: update_lookup
+    json_marshal_mode: canonical
+    app_name: benthos
+    auto_replay_nacks: true
+```
+--
+======
+
+== Prerequisites
+
+- MongoDB version 6 or later
+ifndef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment.
+endif::[]
+ifdef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment. For detailed networking information, including how to set up a VPC peering connection, see xref:networking:index.adoc[Redpanda Cloud Networking].
+endif::[]
+- A sharded MongoDB database or MongoDB cluster running as a replica set
+
+ifndef::env-cloud[]
+== Enable connectivity from cloud-based data sources
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the IP addresses of your Redpanda Connect instances to your firewall rules.
+
+endif::[]
+
+ifdef::env-cloud[]
+
+== Enable connectivity from cloud-based data sources (BYOC)
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the NAT Gateway IP address of your Redpanda cluster to the allowlist of your data source.
+
+endif::[]
+
+== Data capture method
+
+The `mongodb_cdc` input uses https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data changes, which does not propagate _all_ changes to Redpanda Connect. To capture all changes in a MongoDB cluster, including deletions, enable pre- and post-image saving for the cluster and <<database-collections, required collections>>. For more information, see <<document-mode, `document_mode` options>> and the https://www.mongodb.com/docs/manual/changeStreams/#change-streams-with-document-pre--and-post-images[MongoDB documentation].
+
+
+== Data replication
+
+Redpanda Connect allows you to specify which <<collections,database collections>> in your source database to receive changes from.
+
+You can also run the `mongodb_cdc` input in one of two modes, depending on whether you need a snapshot of existing data before streaming updates.
+
+- Snapshot mode: Redpanda Connect first captures a snapshot of all data in the selected collections and streams the contents before processing changes from the last recorded https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^] position.
+- Streaming mode: Redpanda Connect skips the snapshot and processes only the most recent data changes, starting from the latest oplog position.
+
+=== Snapshot mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `true`, Redpanda Connect connects to your MongoDB database and does the following to capture a snapshot of all data in the selected collections:
+
+//TODO should include parallel processing of snapshots and document_mode
+
+NOTE: If the pipeline restarts during this process, Redpanda Connect must start the snapshot capture from scratch to store the current oplog position in the <<checkpoint_cache,`checkpoint_cache`>>.
+
+//TODO
+
+Finally, the input uses the stored oplog position to catch up with changes that occurred during snapshot processing.
+
+=== Streaming mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `false`, Redpanda Connect connects to your MongoDB database and starts processing data changes from the latest oplog position. If the pipeline restarts, Redpanda Connect resumes processing updates from the last oplog position written to the <<checkpoint_cache,`checkpoint_cache`>>.
+
+== Data mappings
+
+The following table shows how selected MongoDB data types are mapped to data types supported in Redpanda Connect. All other data types are mapped to string values.
+
+|===
+| MongoDB data type | Bloblang value
+
+| Placeholder
+| Placeholder
+
+|===
+
+== Metadata
+
+This input adds the following metadata fields to each message:
+
+- `operation`: The type of data change that generated the message: `read`, `create`, `update`, `replace`, `delete`, `update`. A `read` operation occurs when the initial snapshot of the database is processed.
+- `collection`: The name of the collection from which the message originated.
+- `operation_time`: The time the data change was written to the https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^].
+
+== Fields
+
+=== `url`
+
+The URL of the target MongoDB server.
+
+*Type*: `string`
+
+*Default*: `""`
+
+```yml
+# Examples
+
+url: mongodb://localhost:27017
+```
+
+=== `database`
+
+The name of the MongoDB database to stream changes from.
+
+*Type*: `string`
+
+*Default*: `""`
+
+
+=== `username`
+
+The username to connect to the database.
+
+*Type*: `string`
+
+*Default*: `""`
+
+=== `password`
+
+The password to connect to the database.
+
+include::components:partial$secret_warning.adoc[]
+
+*Type*: `string`
+
+*Default*: `""`
+
+
+=== `collections`
+
+A list of collections to stream changes from. Specify each collection name as a separate item.
+
+*Type*: `array`
+
+*Default*: `[]`
+
+```yml
+# Examples
+
+collections:
+  - orders
+  - customers
+  - inventory
+```
+
+=== `checkpoint_cache`
+
+Specify a `cache` resource to store the oplog position for the most recent data update streamed to Redpanda Connect. After a restart, Redpanda Connect can continue processing changes from this position, avoiding the need to reprocess all collection updates.
+
+*Type*: `string`
+
+*Default*: `""`
+
+```yml
+# Examples
+
+input:
+  mongodb_cdc:
+    url: mongodb://localhost:27017
+    collections: [my_collection]
+    checkpoint_cache: "my_cdc_cache"
+cache_resources:
+  - label: "my_cdc_cache"
+    redis:
+      url: redis://:6379
+```
+
+=== `checkpoint_key`
+
+The key identifier used to store the oplog position in <<checkpoint_cache,`checkpoint_cache`>>. If you have multiple `mongodb_cdc` inputs sharing the same cache, you can provide an alternative key.
+
+
+*Type*: `string`
+
+*Default*: `mongodb_cdc_checkpoint`
+
+
+=== `checkpoint_interval`
+
+The interval between writing checkpoints to the cache.
+
+*Type*: `string`
+
+*Default*: `5s`
+
+=== `checkpoint_limit`
+
+The maximum number of messages that this input can process at a given time. Increasing this limit enables parallel processing, and batching at the output level. To preserve at-least-once guarantees, any given oplog position is not acknowledged until all messages under that offset are delivered.
+
+*Type*: `int`
+
+*Default*: `1000`
+
+
+=== `read_batch_size`
+
+The number of documents to fetch in each message batch from MongoDB.
+
+
+*Type*: `int`
+
+*Default*: `1000`
+
+
+=== `read_max_wait`
+
+The maximum duration MongoDB waits to accumulate the <<read-batch-size, `read_batch_size`>> documents on a change stream before returning the batch to Redpanda Connect.
+
+*Type*: `string`
+
+*Default*: `1s`
+
+=== `stream_snapshot`
+
+When set to `true`, this input streams a snapshot of all existing data in the source collections before streaming data changes.
+
+*Type*: `bool`
+
+*Default*: `false`
+
+```yml
+# Examples
+
+stream_snapshot: true
+```
+
+=== `snapshot_parallelism`
+
+Specifies the number of connections to use when reading the initial snapshot from one or more collections. Increase this number to enable parallel processing of the snapshot.
+
+This feature uses the `$splitVector` command to split snapshot data into chunks for more efficient processing.
+
+*Type*: `int`
+
+*Default*: `1`
+
+=== `snapshot_auto_bucket_sharding`
+
+Uses the https://www.mongodb.com/docs/manual/reference/operator/aggregation/bucketAuto/[`$bucketAuto`^] command instead of the default, `$splitVector`, to split the snapshot data into chunks for processing. This is required for environments, such as MongoDB Atlas, where privileged access is restricted. To enable parallel processing in these environments:
+
+- Set `snapshot_auto_bucket_sharding` to `true`.
+- Increase `snapshot_parallelism` to a value greater than `1`.
+
+*Type*: `bool`
+
+*Default*: `false`
+
+=== `document_mode`
+
+The mode in which MongoDB emits document changes to Redpanda Connect, specifically updates and deletes.
+
+[cols="1,2"]
+|===
+| Option | Mode description
+
+| `partial_update`
+a| Update operations contain only a description of the changes made to the document, using the following schema:
+
+[,bash,role="no-placeholders"]
+----
+{
+  "_id": "<document_id>",
+  "operations": [
+    # type == set: Indicates that a value was updated. For example:
+    # root.name."address.city" = "New York"
+    {"path": ["name", "address.city"], "type": "set", "value": "New York"},
+
+    # type == unset: Indicates that a value was deleted. For example:
+    # root.phone = deleted()
+    {"path": ["phone"], "type": "unset", "value": null},
+
+    # type == truncatedArray: Indicates that an array was truncated to a specified number of elements. For example:
+    # root.items = this.items.slice(2)
+    {"path": ["items"], "type": "truncatedArray", "value": 2}
+  ]
+}
+----
+
+| `pre_and_post_images`
+a| Emits full documents for update and delete operations by using pre- and post-image collections.  
+For setup and configuration details, see the https://www.mongodb.com/docs/manual/changeStreams/#change-streams-with-document-pre--and-post-images[MongoDB documentation^].
+
+| `update_lookup`
+a| Emits full documents for insert, replace, and update operations, but only the `_id` field for delete operations.  
+This corresponds to the `updateLookup` option.  
+For more information, see the https://www.mongodb.com/docs/manual/changeStreams/#std-label-change-streams-updateLookup[MongoDB documentation^].
+
+|===
+
+=== `json_marshal_mode`
+
+Controls how MongoDB formats messages sent to Redpanda Connect.
+
+*Type*: `string`
+
+*Default*: `canonical`
+
+|===
+| Option | Description
+
+| `canonical`
+| A string format that preserves type information at the expense of readability and interoperability. Conversion from canonical to Binary JSON (BSON) generally retains type information, except in specific cases.
path: modules/components/pages/inputs/mongodb_cdc.adoc
user: login: asimms41
body: I'm not sure when this conversion from canonical to BSON occurs. When a message is emitted from MongoDB? And then there is a conversion from BSON to JSON when it reaches RPCN?
created_at: 2025-03-06T18:17:21Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,393 @@
+= mongodb_cdc
+// tag::single-source[]
+:type: input
+:page-beta: true
+:categories: ["Services"]
+
+component_type_dropdown::[]
+
+Streams data changes from a MongoDB replica set, using MongoDB's https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data updates.
+
+ifndef::env-cloud[]
+Introduced in version 4.48.1.
+endif::[]
+
+[tabs]
+======
+Common::
++
+--
+
+```yml
+# Configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: "" # Optional
+    password: "" # Optional
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    auto_replay_nacks: true
+```
+
+--
+Advanced::
++
+--
+
+```yml
+# All configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: ""
+    password: ""
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    snapshot_auto_bucket_sharding: false
+    document_mode: update_lookup
+    json_marshal_mode: canonical
+    app_name: benthos
+    auto_replay_nacks: true
+```
+--
+======
+
+== Prerequisites
+
+- MongoDB version 6 or later
+ifndef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment.
+endif::[]
+ifdef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment. For detailed networking information, including how to set up a VPC peering connection, see xref:networking:index.adoc[Redpanda Cloud Networking].
+endif::[]
+- A sharded MongoDB database or MongoDB cluster running as a replica set
+
+ifndef::env-cloud[]
+== Enable connectivity from cloud-based data sources
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the IP addresses of your Redpanda Connect instances to your firewall rules.
+
+endif::[]
+
+ifdef::env-cloud[]
+
+== Enable connectivity from cloud-based data sources (BYOC)
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the NAT Gateway IP address of your Redpanda cluster to the allowlist of your data source.
+
+endif::[]
+
+== Data capture method
+
+The `mongodb_cdc` input uses https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data changes, which does not propagate _all_ changes to Redpanda Connect. To capture all changes in a MongoDB cluster, including deletions, enable pre- and post-image saving for the cluster and <<database-collections, required collections>>. For more information, see <<document-mode, `document_mode` options>> and the https://www.mongodb.com/docs/manual/changeStreams/#change-streams-with-document-pre--and-post-images[MongoDB documentation].
+
+
+== Data replication
+
+Redpanda Connect allows you to specify which <<collections,database collections>> in your source database to receive changes from.
+
+You can also run the `mongodb_cdc` input in one of two modes, depending on whether you need a snapshot of existing data before streaming updates.
+
+- Snapshot mode: Redpanda Connect first captures a snapshot of all data in the selected collections and streams the contents before processing changes from the last recorded https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^] position.
+- Streaming mode: Redpanda Connect skips the snapshot and processes only the most recent data changes, starting from the latest oplog position.
+
+=== Snapshot mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `true`, Redpanda Connect connects to your MongoDB database and does the following to capture a snapshot of all data in the selected collections:
+
+//TODO should include parallel processing of snapshots and document_mode
+
+NOTE: If the pipeline restarts during this process, Redpanda Connect must start the snapshot capture from scratch to store the current oplog position in the <<checkpoint_cache,`checkpoint_cache`>>.
+
+//TODO
+
+Finally, the input uses the stored oplog position to catch up with changes that occurred during snapshot processing.
+
+=== Streaming mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `false`, Redpanda Connect connects to your MongoDB database and starts processing data changes from the latest oplog position. If the pipeline restarts, Redpanda Connect resumes processing updates from the last oplog position written to the <<checkpoint_cache,`checkpoint_cache`>>.
+
+== Data mappings
+
+The following table shows how selected MongoDB data types are mapped to data types supported in Redpanda Connect. All other data types are mapped to string values.
+
+|===
+| MongoDB data type | Bloblang value
+
+| Placeholder
+| Placeholder
+
+|===
+
+== Metadata
+
+This input adds the following metadata fields to each message:
+
+- `operation`: The type of data change that generated the message: `read`, `create`, `update`, `replace`, `delete`, `update`. A `read` operation occurs when the initial snapshot of the database is processed.
+- `collection`: The name of the collection from which the message originated.
+- `operation_time`: The time the data change was written to the https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^].
+
+== Fields
+
+=== `url`
+
+The URL of the target MongoDB server.
+
+*Type*: `string`
+
+*Default*: `""`
+
+```yml
+# Examples
+
+url: mongodb://localhost:27017
+```
+
+=== `database`
+
+The name of the MongoDB database to stream changes from.
+
+*Type*: `string`
+
+*Default*: `""`
+
+
+=== `username`
+
+The username to connect to the database.
+
+*Type*: `string`
+
+*Default*: `""`
+
+=== `password`
+
+The password to connect to the database.
+
+include::components:partial$secret_warning.adoc[]
+
+*Type*: `string`
+
+*Default*: `""`
+
+
+=== `collections`
+
+A list of collections to stream changes from. Specify each collection name as a separate item.
+
+*Type*: `array`
+
+*Default*: `[]`
+
+```yml
+# Examples
+
+collections:
+  - orders
+  - customers
+  - inventory
+```
+
+=== `checkpoint_cache`
+
+Specify a `cache` resource to store the oplog position for the most recent data update streamed to Redpanda Connect. After a restart, Redpanda Connect can continue processing changes from this position, avoiding the need to reprocess all collection updates.
+
+*Type*: `string`
+
+*Default*: `""`
+
+```yml
+# Examples
+
+input:
+  mongodb_cdc:
+    url: mongodb://localhost:27017
+    collections: [my_collection]
+    checkpoint_cache: "my_cdc_cache"
+cache_resources:
+  - label: "my_cdc_cache"
+    redis:
+      url: redis://:6379
+```
+
+=== `checkpoint_key`
+
+The key identifier used to store the oplog position in <<checkpoint_cache,`checkpoint_cache`>>. If you have multiple `mongodb_cdc` inputs sharing the same cache, you can provide an alternative key.
+
+
+*Type*: `string`
+
+*Default*: `mongodb_cdc_checkpoint`
+
+
+=== `checkpoint_interval`
+
+The interval between writing checkpoints to the cache.
+
+*Type*: `string`
+
+*Default*: `5s`
+
+=== `checkpoint_limit`
+
+The maximum number of messages that this input can process at a given time. Increasing this limit enables parallel processing, and batching at the output level. To preserve at-least-once guarantees, any given oplog position is not acknowledged until all messages under that offset are delivered.
+
+*Type*: `int`
+
+*Default*: `1000`
+
+
+=== `read_batch_size`
+
+The number of documents to fetch in each message batch from MongoDB.
path: modules/components/pages/inputs/mongodb_cdc.adoc
user: login: asimms41
body: Is this just for the initial snapshot read?
created_at: 2025-03-06T18:18:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,393 @@
+= mongodb_cdc
+// tag::single-source[]
+:type: input
+:page-beta: true
+:categories: ["Services"]
+
+component_type_dropdown::[]
+
+Streams data changes from a MongoDB replica set, using MongoDB's https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data updates.
+
+ifndef::env-cloud[]
+Introduced in version 4.48.1.
+endif::[]
+
+[tabs]
+======
+Common::
++
+--
+
+```yml
+# Configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: "" # Optional
+    password: "" # Optional
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    auto_replay_nacks: true
+```
+
+--
+Advanced::
++
+--
+
+```yml
+# All configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: ""
+    password: ""
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    snapshot_auto_bucket_sharding: false
+    document_mode: update_lookup
+    json_marshal_mode: canonical
+    app_name: benthos
+    auto_replay_nacks: true
+```
+--
+======
+
+== Prerequisites
+
+- MongoDB version 6 or later
+ifndef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment.
+endif::[]
+ifdef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment. For detailed networking information, including how to set up a VPC peering connection, see xref:networking:index.adoc[Redpanda Cloud Networking].
+endif::[]
+- A sharded MongoDB database or MongoDB cluster running as a replica set
path: modules/components/pages/inputs/mongodb_cdc.adoc
user: login: rockwotj
body: maybe link to similar items as this post: https://www.mongodb.com/resources/products/capabilities/change-streams see the "Availability of MongoDB Change Streams" section
created_at: 2025-03-07T08:01:36Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,393 @@
+= mongodb_cdc
+// tag::single-source[]
+:type: input
+:page-beta: true
+:categories: ["Services"]
+
+component_type_dropdown::[]
+
+Streams data changes from a MongoDB replica set, using MongoDB's https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data updates.
+
+ifndef::env-cloud[]
+Introduced in version 4.48.1.
+endif::[]
+
+[tabs]
+======
+Common::
++
+--
+
+```yml
+# Configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: "" # Optional
+    password: "" # Optional
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    auto_replay_nacks: true
+```
+
+--
+Advanced::
++
+--
+
+```yml
+# All configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: ""
+    password: ""
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    snapshot_auto_bucket_sharding: false
+    document_mode: update_lookup
+    json_marshal_mode: canonical
+    app_name: benthos
+    auto_replay_nacks: true
+```
+--
+======
+
+== Prerequisites
+
+- MongoDB version 6 or later
+ifndef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment.
+endif::[]
+ifdef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment. For detailed networking information, including how to set up a VPC peering connection, see xref:networking:index.adoc[Redpanda Cloud Networking].
+endif::[]
+- A sharded MongoDB database or MongoDB cluster running as a replica set
+
+ifndef::env-cloud[]
+== Enable connectivity from cloud-based data sources
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the IP addresses of your Redpanda Connect instances to your firewall rules.
+
+endif::[]
+
+ifdef::env-cloud[]
+
+== Enable connectivity from cloud-based data sources (BYOC)
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the NAT Gateway IP address of your Redpanda cluster to the allowlist of your data source.
+
+endif::[]
+
+== Data capture method
+
+The `mongodb_cdc` input uses https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data changes, which does not propagate _all_ changes to Redpanda Connect. To capture all changes in a MongoDB cluster, including deletions, enable pre- and post-image saving for the cluster and <<database-collections, required collections>>. For more information, see <<document-mode, `document_mode` options>> and the https://www.mongodb.com/docs/manual/changeStreams/#change-streams-with-document-pre--and-post-images[MongoDB documentation].
+
+
+== Data replication
+
+Redpanda Connect allows you to specify which <<collections,database collections>> in your source database to receive changes from.
+
+You can also run the `mongodb_cdc` input in one of two modes, depending on whether you need a snapshot of existing data before streaming updates.
+
+- Snapshot mode: Redpanda Connect first captures a snapshot of all data in the selected collections and streams the contents before processing changes from the last recorded https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^] position.
+- Streaming mode: Redpanda Connect skips the snapshot and processes only the most recent data changes, starting from the latest oplog position.
+
+=== Snapshot mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `true`, Redpanda Connect connects to your MongoDB database and does the following to capture a snapshot of all data in the selected collections:
+
+//TODO should include parallel processing of snapshots and document_mode
path: modules/components/pages/inputs/mongodb_cdc.adoc
user: login: rockwotj
body: Here is the algorithm:

- Record the latest oplog position 
- determine shards using the configured strategy. this is based on bucket auto sharding true/false.

If false (default) we use the internal splitVector command to get shards for each collection then read all collections in parallel using up the max snapshot parallelism workers.

If true we use a autoBucket aggregation to compute the shards and then everything else is the same.

Once we complete reading the snapshot we start the change stream from the earlier recorded oplog position 
created_at: 2025-03-07T08:14:59Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,393 @@
+= mongodb_cdc
+// tag::single-source[]
+:type: input
+:page-beta: true
+:categories: ["Services"]
+
+component_type_dropdown::[]
+
+Streams data changes from a MongoDB replica set, using MongoDB's https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data updates.
+
+ifndef::env-cloud[]
+Introduced in version 4.48.1.
+endif::[]
+
+[tabs]
+======
+Common::
++
+--
+
+```yml
+# Configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: "" # Optional
+    password: "" # Optional
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    auto_replay_nacks: true
+```
+
+--
+Advanced::
++
+--
+
+```yml
+# All configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: ""
+    password: ""
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    snapshot_auto_bucket_sharding: false
+    document_mode: update_lookup
+    json_marshal_mode: canonical
+    app_name: benthos
+    auto_replay_nacks: true
+```
+--
+======
+
+== Prerequisites
+
+- MongoDB version 6 or later
+ifndef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment.
+endif::[]
+ifdef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment. For detailed networking information, including how to set up a VPC peering connection, see xref:networking:index.adoc[Redpanda Cloud Networking].
+endif::[]
+- A sharded MongoDB database or MongoDB cluster running as a replica set
+
+ifndef::env-cloud[]
+== Enable connectivity from cloud-based data sources
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the IP addresses of your Redpanda Connect instances to your firewall rules.
+
+endif::[]
+
+ifdef::env-cloud[]
+
+== Enable connectivity from cloud-based data sources (BYOC)
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the NAT Gateway IP address of your Redpanda cluster to the allowlist of your data source.
+
+endif::[]
+
+== Data capture method
+
+The `mongodb_cdc` input uses https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data changes, which does not propagate _all_ changes to Redpanda Connect. To capture all changes in a MongoDB cluster, including deletions, enable pre- and post-image saving for the cluster and <<database-collections, required collections>>. For more information, see <<document-mode, `document_mode` options>> and the https://www.mongodb.com/docs/manual/changeStreams/#change-streams-with-document-pre--and-post-images[MongoDB documentation].
+
+
+== Data replication
+
+Redpanda Connect allows you to specify which <<collections,database collections>> in your source database to receive changes from.
+
+You can also run the `mongodb_cdc` input in one of two modes, depending on whether you need a snapshot of existing data before streaming updates.
+
+- Snapshot mode: Redpanda Connect first captures a snapshot of all data in the selected collections and streams the contents before processing changes from the last recorded https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^] position.
+- Streaming mode: Redpanda Connect skips the snapshot and processes only the most recent data changes, starting from the latest oplog position.
+
+=== Snapshot mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `true`, Redpanda Connect connects to your MongoDB database and does the following to capture a snapshot of all data in the selected collections:
+
+//TODO should include parallel processing of snapshots and document_mode
+
+NOTE: If the pipeline restarts during this process, Redpanda Connect must start the snapshot capture from scratch to store the current oplog position in the <<checkpoint_cache,`checkpoint_cache`>>.
+
+//TODO
+
+Finally, the input uses the stored oplog position to catch up with changes that occurred during snapshot processing.
+
+=== Streaming mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `false`, Redpanda Connect connects to your MongoDB database and starts processing data changes from the latest oplog position. If the pipeline restarts, Redpanda Connect resumes processing updates from the last oplog position written to the <<checkpoint_cache,`checkpoint_cache`>>.
+
+== Data mappings
path: modules/components/pages/inputs/mongodb_cdc.adoc
user: login: rockwotj
body: No it's not
created_at: 2025-03-07T08:16:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,393 @@
+= mongodb_cdc
+// tag::single-source[]
+:type: input
+:page-beta: true
+:categories: ["Services"]
+
+component_type_dropdown::[]
+
+Streams data changes from a MongoDB replica set, using MongoDB's https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data updates.
+
+ifndef::env-cloud[]
+Introduced in version 4.48.1.
+endif::[]
+
+[tabs]
+======
+Common::
++
+--
+
+```yml
+# Configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: "" # Optional
+    password: "" # Optional
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    auto_replay_nacks: true
+```
+
+--
+Advanced::
++
+--
+
+```yml
+# All configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: ""
+    password: ""
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    snapshot_auto_bucket_sharding: false
+    document_mode: update_lookup
+    json_marshal_mode: canonical
+    app_name: benthos
+    auto_replay_nacks: true
+```
+--
+======
+
+== Prerequisites
+
+- MongoDB version 6 or later
+ifndef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment.
+endif::[]
+ifdef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment. For detailed networking information, including how to set up a VPC peering connection, see xref:networking:index.adoc[Redpanda Cloud Networking].
+endif::[]
+- A sharded MongoDB database or MongoDB cluster running as a replica set
+
+ifndef::env-cloud[]
+== Enable connectivity from cloud-based data sources
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the IP addresses of your Redpanda Connect instances to your firewall rules.
+
+endif::[]
+
+ifdef::env-cloud[]
+
+== Enable connectivity from cloud-based data sources (BYOC)
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the NAT Gateway IP address of your Redpanda cluster to the allowlist of your data source.
+
+endif::[]
+
+== Data capture method
+
+The `mongodb_cdc` input uses https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data changes, which does not propagate _all_ changes to Redpanda Connect. To capture all changes in a MongoDB cluster, including deletions, enable pre- and post-image saving for the cluster and <<database-collections, required collections>>. For more information, see <<document-mode, `document_mode` options>> and the https://www.mongodb.com/docs/manual/changeStreams/#change-streams-with-document-pre--and-post-images[MongoDB documentation].
+
+
+== Data replication
+
+Redpanda Connect allows you to specify which <<collections,database collections>> in your source database to receive changes from.
+
+You can also run the `mongodb_cdc` input in one of two modes, depending on whether you need a snapshot of existing data before streaming updates.
+
+- Snapshot mode: Redpanda Connect first captures a snapshot of all data in the selected collections and streams the contents before processing changes from the last recorded https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^] position.
+- Streaming mode: Redpanda Connect skips the snapshot and processes only the most recent data changes, starting from the latest oplog position.
+
+=== Snapshot mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `true`, Redpanda Connect connects to your MongoDB database and does the following to capture a snapshot of all data in the selected collections:
+
+//TODO should include parallel processing of snapshots and document_mode
+
+NOTE: If the pipeline restarts during this process, Redpanda Connect must start the snapshot capture from scratch to store the current oplog position in the <<checkpoint_cache,`checkpoint_cache`>>.
+
+//TODO
+
+Finally, the input uses the stored oplog position to catch up with changes that occurred during snapshot processing.
+
+=== Streaming mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `false`, Redpanda Connect connects to your MongoDB database and starts processing data changes from the latest oplog position. If the pipeline restarts, Redpanda Connect resumes processing updates from the last oplog position written to the <<checkpoint_cache,`checkpoint_cache`>>.
+
+== Data mappings
+
+The following table shows how selected MongoDB data types are mapped to data types supported in Redpanda Connect. All other data types are mapped to string values.
+
+|===
+| MongoDB data type | Bloblang value
+
+| Placeholder
+| Placeholder
+
+|===
+
+== Metadata
+
+This input adds the following metadata fields to each message:
+
+- `operation`: The type of data change that generated the message: `read`, `create`, `update`, `replace`, `delete`, `update`. A `read` operation occurs when the initial snapshot of the database is processed.
+- `collection`: The name of the collection from which the message originated.
+- `operation_time`: The time the data change was written to the https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^].
path: modules/components/pages/inputs/mongodb_cdc.adoc
user: login: rockwotj
body: This is a bson timestamp meaning it's in the form of `{"t": <unix seconds>, "i": <sequence number>}`
created_at: 2025-03-07T08:17:13Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,393 @@
+= mongodb_cdc
+// tag::single-source[]
+:type: input
+:page-beta: true
+:categories: ["Services"]
+
+component_type_dropdown::[]
+
+Streams data changes from a MongoDB replica set, using MongoDB's https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data updates.
+
+ifndef::env-cloud[]
+Introduced in version 4.48.1.
+endif::[]
+
+[tabs]
+======
+Common::
++
+--
+
+```yml
+# Configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: "" # Optional
+    password: "" # Optional
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    auto_replay_nacks: true
+```
+
+--
+Advanced::
++
+--
+
+```yml
+# All configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: ""
+    password: ""
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    snapshot_auto_bucket_sharding: false
+    document_mode: update_lookup
+    json_marshal_mode: canonical
+    app_name: benthos
+    auto_replay_nacks: true
+```
+--
+======
+
+== Prerequisites
+
+- MongoDB version 6 or later
+ifndef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment.
+endif::[]
+ifdef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment. For detailed networking information, including how to set up a VPC peering connection, see xref:networking:index.adoc[Redpanda Cloud Networking].
+endif::[]
+- A sharded MongoDB database or MongoDB cluster running as a replica set
+
+ifndef::env-cloud[]
+== Enable connectivity from cloud-based data sources
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the IP addresses of your Redpanda Connect instances to your firewall rules.
+
+endif::[]
+
+ifdef::env-cloud[]
+
+== Enable connectivity from cloud-based data sources (BYOC)
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the NAT Gateway IP address of your Redpanda cluster to the allowlist of your data source.
+
+endif::[]
+
+== Data capture method
+
+The `mongodb_cdc` input uses https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data changes, which does not propagate _all_ changes to Redpanda Connect. To capture all changes in a MongoDB cluster, including deletions, enable pre- and post-image saving for the cluster and <<database-collections, required collections>>. For more information, see <<document-mode, `document_mode` options>> and the https://www.mongodb.com/docs/manual/changeStreams/#change-streams-with-document-pre--and-post-images[MongoDB documentation].
+
+
+== Data replication
+
+Redpanda Connect allows you to specify which <<collections,database collections>> in your source database to receive changes from.
+
+You can also run the `mongodb_cdc` input in one of two modes, depending on whether you need a snapshot of existing data before streaming updates.
+
+- Snapshot mode: Redpanda Connect first captures a snapshot of all data in the selected collections and streams the contents before processing changes from the last recorded https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^] position.
+- Streaming mode: Redpanda Connect skips the snapshot and processes only the most recent data changes, starting from the latest oplog position.
+
+=== Snapshot mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `true`, Redpanda Connect connects to your MongoDB database and does the following to capture a snapshot of all data in the selected collections:
+
+//TODO should include parallel processing of snapshots and document_mode
+
+NOTE: If the pipeline restarts during this process, Redpanda Connect must start the snapshot capture from scratch to store the current oplog position in the <<checkpoint_cache,`checkpoint_cache`>>.
+
+//TODO
+
+Finally, the input uses the stored oplog position to catch up with changes that occurred during snapshot processing.
+
+=== Streaming mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `false`, Redpanda Connect connects to your MongoDB database and starts processing data changes from the latest oplog position. If the pipeline restarts, Redpanda Connect resumes processing updates from the last oplog position written to the <<checkpoint_cache,`checkpoint_cache`>>.
+
+== Data mappings
+
+The following table shows how selected MongoDB data types are mapped to data types supported in Redpanda Connect. All other data types are mapped to string values.
+
+|===
+| MongoDB data type | Bloblang value
+
+| Placeholder
+| Placeholder
+
+|===
+
+== Metadata
+
+This input adds the following metadata fields to each message:
+
+- `operation`: The type of data change that generated the message: `read`, `create`, `update`, `replace`, `delete`, `update`. A `read` operation occurs when the initial snapshot of the database is processed.
+- `collection`: The name of the collection from which the message originated.
+- `operation_time`: The time the data change was written to the https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^].
+
+== Fields
+
+=== `url`
+
+The URL of the target MongoDB server.
+
+*Type*: `string`
+
+*Default*: `""`
+
+```yml
+# Examples
+
+url: mongodb://localhost:27017
+```
+
+=== `database`
+
+The name of the MongoDB database to stream changes from.
+
+*Type*: `string`
+
+*Default*: `""`
+
+
+=== `username`
+
+The username to connect to the database.
+
+*Type*: `string`
+
+*Default*: `""`
+
+=== `password`
+
+The password to connect to the database.
+
+include::components:partial$secret_warning.adoc[]
+
+*Type*: `string`
+
+*Default*: `""`
+
+
+=== `collections`
+
+A list of collections to stream changes from. Specify each collection name as a separate item.
+
+*Type*: `array`
+
+*Default*: `[]`
+
+```yml
+# Examples
+
+collections:
+  - orders
+  - customers
+  - inventory
+```
+
+=== `checkpoint_cache`
+
+Specify a `cache` resource to store the oplog position for the most recent data update streamed to Redpanda Connect. After a restart, Redpanda Connect can continue processing changes from this position, avoiding the need to reprocess all collection updates.
+
+*Type*: `string`
+
+*Default*: `""`
+
+```yml
+# Examples
+
+input:
+  mongodb_cdc:
+    url: mongodb://localhost:27017
+    collections: [my_collection]
+    checkpoint_cache: "my_cdc_cache"
+cache_resources:
+  - label: "my_cdc_cache"
+    redis:
+      url: redis://:6379
+```
+
+=== `checkpoint_key`
+
+The key identifier used to store the oplog position in <<checkpoint_cache,`checkpoint_cache`>>. If you have multiple `mongodb_cdc` inputs sharing the same cache, you can provide an alternative key.
+
+
+*Type*: `string`
+
+*Default*: `mongodb_cdc_checkpoint`
+
+
+=== `checkpoint_interval`
+
+The interval between writing checkpoints to the cache.
+
+*Type*: `string`
+
+*Default*: `5s`
+
+=== `checkpoint_limit`
+
+The maximum number of messages that this input can process at a given time. Increasing this limit enables parallel processing, and batching at the output level. To preserve at-least-once guarantees, any given oplog position is not acknowledged until all messages under that offset are delivered.
path: modules/components/pages/inputs/mongodb_cdc.adoc
user: login: rockwotj
body: I would not say "input would process at a given time" but something like: "number of inflight messages from this input at a given time"
created_at: 2025-03-07T08:47:45Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,393 @@
+= mongodb_cdc
+// tag::single-source[]
+:type: input
+:page-beta: true
+:categories: ["Services"]
+
+component_type_dropdown::[]
+
+Streams data changes from a MongoDB replica set, using MongoDB's https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data updates.
+
+ifndef::env-cloud[]
+Introduced in version 4.48.1.
+endif::[]
+
+[tabs]
+======
+Common::
++
+--
+
+```yml
+# Configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: "" # Optional
+    password: "" # Optional
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    auto_replay_nacks: true
+```
+
+--
+Advanced::
++
+--
+
+```yml
+# All configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: ""
+    password: ""
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    snapshot_auto_bucket_sharding: false
+    document_mode: update_lookup
+    json_marshal_mode: canonical
+    app_name: benthos
+    auto_replay_nacks: true
+```
+--
+======
+
+== Prerequisites
+
+- MongoDB version 6 or later
+ifndef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment.
+endif::[]
+ifdef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment. For detailed networking information, including how to set up a VPC peering connection, see xref:networking:index.adoc[Redpanda Cloud Networking].
+endif::[]
+- A sharded MongoDB database or MongoDB cluster running as a replica set
+
+ifndef::env-cloud[]
+== Enable connectivity from cloud-based data sources
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the IP addresses of your Redpanda Connect instances to your firewall rules.
+
+endif::[]
+
+ifdef::env-cloud[]
+
+== Enable connectivity from cloud-based data sources (BYOC)
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the NAT Gateway IP address of your Redpanda cluster to the allowlist of your data source.
+
+endif::[]
+
+== Data capture method
+
+The `mongodb_cdc` input uses https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data changes, which does not propagate _all_ changes to Redpanda Connect. To capture all changes in a MongoDB cluster, including deletions, enable pre- and post-image saving for the cluster and <<database-collections, required collections>>. For more information, see <<document-mode, `document_mode` options>> and the https://www.mongodb.com/docs/manual/changeStreams/#change-streams-with-document-pre--and-post-images[MongoDB documentation].
+
+
+== Data replication
+
+Redpanda Connect allows you to specify which <<collections,database collections>> in your source database to receive changes from.
+
+You can also run the `mongodb_cdc` input in one of two modes, depending on whether you need a snapshot of existing data before streaming updates.
+
+- Snapshot mode: Redpanda Connect first captures a snapshot of all data in the selected collections and streams the contents before processing changes from the last recorded https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^] position.
+- Streaming mode: Redpanda Connect skips the snapshot and processes only the most recent data changes, starting from the latest oplog position.
+
+=== Snapshot mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `true`, Redpanda Connect connects to your MongoDB database and does the following to capture a snapshot of all data in the selected collections:
+
+//TODO should include parallel processing of snapshots and document_mode
+
+NOTE: If the pipeline restarts during this process, Redpanda Connect must start the snapshot capture from scratch to store the current oplog position in the <<checkpoint_cache,`checkpoint_cache`>>.
+
+//TODO
+
+Finally, the input uses the stored oplog position to catch up with changes that occurred during snapshot processing.
+
+=== Streaming mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `false`, Redpanda Connect connects to your MongoDB database and starts processing data changes from the latest oplog position. If the pipeline restarts, Redpanda Connect resumes processing updates from the last oplog position written to the <<checkpoint_cache,`checkpoint_cache`>>.
+
+== Data mappings
+
+The following table shows how selected MongoDB data types are mapped to data types supported in Redpanda Connect. All other data types are mapped to string values.
+
+|===
+| MongoDB data type | Bloblang value
+
+| Placeholder
+| Placeholder
+
+|===
+
+== Metadata
+
+This input adds the following metadata fields to each message:
+
+- `operation`: The type of data change that generated the message: `read`, `create`, `update`, `replace`, `delete`, `update`. A `read` operation occurs when the initial snapshot of the database is processed.
+- `collection`: The name of the collection from which the message originated.
+- `operation_time`: The time the data change was written to the https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^].
+
+== Fields
+
+=== `url`
+
+The URL of the target MongoDB server.
+
+*Type*: `string`
+
+*Default*: `""`
+
+```yml
+# Examples
+
+url: mongodb://localhost:27017
+```
+
+=== `database`
+
+The name of the MongoDB database to stream changes from.
+
+*Type*: `string`
+
+*Default*: `""`
+
+
+=== `username`
+
+The username to connect to the database.
+
+*Type*: `string`
+
+*Default*: `""`
+
+=== `password`
+
+The password to connect to the database.
+
+include::components:partial$secret_warning.adoc[]
+
+*Type*: `string`
+
+*Default*: `""`
+
+
+=== `collections`
+
+A list of collections to stream changes from. Specify each collection name as a separate item.
+
+*Type*: `array`
+
+*Default*: `[]`
+
+```yml
+# Examples
+
+collections:
+  - orders
+  - customers
+  - inventory
+```
+
+=== `checkpoint_cache`
+
+Specify a `cache` resource to store the oplog position for the most recent data update streamed to Redpanda Connect. After a restart, Redpanda Connect can continue processing changes from this position, avoiding the need to reprocess all collection updates.
+
+*Type*: `string`
+
+*Default*: `""`
+
+```yml
+# Examples
+
+input:
+  mongodb_cdc:
+    url: mongodb://localhost:27017
+    collections: [my_collection]
+    checkpoint_cache: "my_cdc_cache"
+cache_resources:
+  - label: "my_cdc_cache"
+    redis:
+      url: redis://:6379
+```
+
+=== `checkpoint_key`
+
+The key identifier used to store the oplog position in <<checkpoint_cache,`checkpoint_cache`>>. If you have multiple `mongodb_cdc` inputs sharing the same cache, you can provide an alternative key.
+
+
+*Type*: `string`
+
+*Default*: `mongodb_cdc_checkpoint`
+
+
+=== `checkpoint_interval`
+
+The interval between writing checkpoints to the cache.
+
+*Type*: `string`
+
+*Default*: `5s`
+
+=== `checkpoint_limit`
+
+The maximum number of messages that this input can process at a given time. Increasing this limit enables parallel processing, and batching at the output level. To preserve at-least-once guarantees, any given oplog position is not acknowledged until all messages under that offset are delivered.
+
+*Type*: `int`
+
+*Default*: `1000`
+
+
+=== `read_batch_size`
+
+The number of documents to fetch in each message batch from MongoDB.
path: modules/components/pages/inputs/mongodb_cdc.adoc
user: login: rockwotj
body: No, for the streaming phase mongo waits up to max time or batch size before sending us new data 
created_at: 2025-03-07T08:47:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,393 @@
+= mongodb_cdc
+// tag::single-source[]
+:type: input
+:page-beta: true
+:categories: ["Services"]
+
+component_type_dropdown::[]
+
+Streams data changes from a MongoDB replica set, using MongoDB's https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data updates.
+
+ifndef::env-cloud[]
+Introduced in version 4.48.1.
+endif::[]
+
+[tabs]
+======
+Common::
++
+--
+
+```yml
+# Configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: "" # Optional
+    password: "" # Optional
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    auto_replay_nacks: true
+```
+
+--
+Advanced::
++
+--
+
+```yml
+# All configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: ""
+    password: ""
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    snapshot_auto_bucket_sharding: false
+    document_mode: update_lookup
+    json_marshal_mode: canonical
+    app_name: benthos
+    auto_replay_nacks: true
+```
+--
+======
+
+== Prerequisites
+
+- MongoDB version 6 or later
+ifndef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment.
+endif::[]
+ifdef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment. For detailed networking information, including how to set up a VPC peering connection, see xref:networking:index.adoc[Redpanda Cloud Networking].
+endif::[]
+- A sharded MongoDB database or MongoDB cluster running as a replica set
+
+ifndef::env-cloud[]
+== Enable connectivity from cloud-based data sources
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the IP addresses of your Redpanda Connect instances to your firewall rules.
+
+endif::[]
+
+ifdef::env-cloud[]
+
+== Enable connectivity from cloud-based data sources (BYOC)
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the NAT Gateway IP address of your Redpanda cluster to the allowlist of your data source.
+
+endif::[]
+
+== Data capture method
+
+The `mongodb_cdc` input uses https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data changes, which does not propagate _all_ changes to Redpanda Connect. To capture all changes in a MongoDB cluster, including deletions, enable pre- and post-image saving for the cluster and <<database-collections, required collections>>. For more information, see <<document-mode, `document_mode` options>> and the https://www.mongodb.com/docs/manual/changeStreams/#change-streams-with-document-pre--and-post-images[MongoDB documentation].
+
+
+== Data replication
+
+Redpanda Connect allows you to specify which <<collections,database collections>> in your source database to receive changes from.
+
+You can also run the `mongodb_cdc` input in one of two modes, depending on whether you need a snapshot of existing data before streaming updates.
+
+- Snapshot mode: Redpanda Connect first captures a snapshot of all data in the selected collections and streams the contents before processing changes from the last recorded https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^] position.
+- Streaming mode: Redpanda Connect skips the snapshot and processes only the most recent data changes, starting from the latest oplog position.
+
+=== Snapshot mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `true`, Redpanda Connect connects to your MongoDB database and does the following to capture a snapshot of all data in the selected collections:
+
+//TODO should include parallel processing of snapshots and document_mode
+
+NOTE: If the pipeline restarts during this process, Redpanda Connect must start the snapshot capture from scratch to store the current oplog position in the <<checkpoint_cache,`checkpoint_cache`>>.
+
+//TODO
+
+Finally, the input uses the stored oplog position to catch up with changes that occurred during snapshot processing.
+
+=== Streaming mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `false`, Redpanda Connect connects to your MongoDB database and starts processing data changes from the latest oplog position. If the pipeline restarts, Redpanda Connect resumes processing updates from the last oplog position written to the <<checkpoint_cache,`checkpoint_cache`>>.
+
+== Data mappings
+
+The following table shows how selected MongoDB data types are mapped to data types supported in Redpanda Connect. All other data types are mapped to string values.
+
+|===
+| MongoDB data type | Bloblang value
+
+| Placeholder
+| Placeholder
+
+|===
+
+== Metadata
+
+This input adds the following metadata fields to each message:
+
+- `operation`: The type of data change that generated the message: `read`, `create`, `update`, `replace`, `delete`, `update`. A `read` operation occurs when the initial snapshot of the database is processed.
+- `collection`: The name of the collection from which the message originated.
+- `operation_time`: The time the data change was written to the https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^].
+
+== Fields
+
+=== `url`
+
+The URL of the target MongoDB server.
+
+*Type*: `string`
+
+*Default*: `""`
+
+```yml
+# Examples
+
+url: mongodb://localhost:27017
+```
+
+=== `database`
+
+The name of the MongoDB database to stream changes from.
+
+*Type*: `string`
+
+*Default*: `""`
+
+
+=== `username`
+
+The username to connect to the database.
+
+*Type*: `string`
+
+*Default*: `""`
+
+=== `password`
+
+The password to connect to the database.
+
+include::components:partial$secret_warning.adoc[]
+
+*Type*: `string`
+
+*Default*: `""`
+
+
+=== `collections`
+
+A list of collections to stream changes from. Specify each collection name as a separate item.
+
+*Type*: `array`
+
+*Default*: `[]`
+
+```yml
+# Examples
+
+collections:
+  - orders
+  - customers
+  - inventory
+```
+
+=== `checkpoint_cache`
+
+Specify a `cache` resource to store the oplog position for the most recent data update streamed to Redpanda Connect. After a restart, Redpanda Connect can continue processing changes from this position, avoiding the need to reprocess all collection updates.
+
+*Type*: `string`
+
+*Default*: `""`
+
+```yml
+# Examples
+
+input:
+  mongodb_cdc:
+    url: mongodb://localhost:27017
+    collections: [my_collection]
+    checkpoint_cache: "my_cdc_cache"
+cache_resources:
+  - label: "my_cdc_cache"
+    redis:
+      url: redis://:6379
+```
+
+=== `checkpoint_key`
+
+The key identifier used to store the oplog position in <<checkpoint_cache,`checkpoint_cache`>>. If you have multiple `mongodb_cdc` inputs sharing the same cache, you can provide an alternative key.
+
+
+*Type*: `string`
+
+*Default*: `mongodb_cdc_checkpoint`
+
+
+=== `checkpoint_interval`
+
+The interval between writing checkpoints to the cache.
+
+*Type*: `string`
+
+*Default*: `5s`
+
+=== `checkpoint_limit`
+
+The maximum number of messages that this input can process at a given time. Increasing this limit enables parallel processing, and batching at the output level. To preserve at-least-once guarantees, any given oplog position is not acknowledged until all messages under that offset are delivered.
+
+*Type*: `int`
+
+*Default*: `1000`
+
+
+=== `read_batch_size`
+
+The number of documents to fetch in each message batch from MongoDB.
+
+
+*Type*: `int`
+
+*Default*: `1000`
+
+
+=== `read_max_wait`
+
+The maximum duration MongoDB waits to accumulate the <<read-batch-size, `read_batch_size`>> documents on a change stream before returning the batch to Redpanda Connect.
+
+*Type*: `string`
+
+*Default*: `1s`
+
+=== `stream_snapshot`
+
+When set to `true`, this input streams a snapshot of all existing data in the source collections before streaming data changes.
+
+*Type*: `bool`
+
+*Default*: `false`
+
+```yml
+# Examples
+
+stream_snapshot: true
+```
+
+=== `snapshot_parallelism`
+
+Specifies the number of connections to use when reading the initial snapshot from one or more collections. Increase this number to enable parallel processing of the snapshot.
+
+This feature uses the `$splitVector` command to split snapshot data into chunks for more efficient processing.
+
+*Type*: `int`
+
+*Default*: `1`
+
+=== `snapshot_auto_bucket_sharding`
+
+Uses the https://www.mongodb.com/docs/manual/reference/operator/aggregation/bucketAuto/[`$bucketAuto`^] command instead of the default, `$splitVector`, to split the snapshot data into chunks for processing. This is required for environments, such as MongoDB Atlas, where privileged access is restricted. To enable parallel processing in these environments:
+
+- Set `snapshot_auto_bucket_sharding` to `true`.
path: modules/components/pages/inputs/mongodb_cdc.adoc
user: login: rockwotj
body: this should be stream_snapshot?
created_at: 2025-03-07T08:50:06Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,393 @@
+= mongodb_cdc
+// tag::single-source[]
+:type: input
+:page-beta: true
+:categories: ["Services"]
+
+component_type_dropdown::[]
+
+Streams data changes from a MongoDB replica set, using MongoDB's https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data updates.
+
+ifndef::env-cloud[]
+Introduced in version 4.48.1.
+endif::[]
+
+[tabs]
+======
+Common::
++
+--
+
+```yml
+# Configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: "" # Optional
+    password: "" # Optional
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    auto_replay_nacks: true
+```
+
+--
+Advanced::
++
+--
+
+```yml
+# All configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: ""
+    password: ""
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    snapshot_auto_bucket_sharding: false
+    document_mode: update_lookup
+    json_marshal_mode: canonical
+    app_name: benthos
+    auto_replay_nacks: true
+```
+--
+======
+
+== Prerequisites
+
+- MongoDB version 6 or later
+ifndef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment.
+endif::[]
+ifdef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment. For detailed networking information, including how to set up a VPC peering connection, see xref:networking:index.adoc[Redpanda Cloud Networking].
+endif::[]
+- A sharded MongoDB database or MongoDB cluster running as a replica set
+
+ifndef::env-cloud[]
+== Enable connectivity from cloud-based data sources
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the IP addresses of your Redpanda Connect instances to your firewall rules.
+
+endif::[]
+
+ifdef::env-cloud[]
+
+== Enable connectivity from cloud-based data sources (BYOC)
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the NAT Gateway IP address of your Redpanda cluster to the allowlist of your data source.
+
+endif::[]
+
+== Data capture method
+
+The `mongodb_cdc` input uses https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data changes, which does not propagate _all_ changes to Redpanda Connect. To capture all changes in a MongoDB cluster, including deletions, enable pre- and post-image saving for the cluster and <<database-collections, required collections>>. For more information, see <<document-mode, `document_mode` options>> and the https://www.mongodb.com/docs/manual/changeStreams/#change-streams-with-document-pre--and-post-images[MongoDB documentation].
+
+
+== Data replication
+
+Redpanda Connect allows you to specify which <<collections,database collections>> in your source database to receive changes from.
+
+You can also run the `mongodb_cdc` input in one of two modes, depending on whether you need a snapshot of existing data before streaming updates.
+
+- Snapshot mode: Redpanda Connect first captures a snapshot of all data in the selected collections and streams the contents before processing changes from the last recorded https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^] position.
+- Streaming mode: Redpanda Connect skips the snapshot and processes only the most recent data changes, starting from the latest oplog position.
+
+=== Snapshot mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `true`, Redpanda Connect connects to your MongoDB database and does the following to capture a snapshot of all data in the selected collections:
+
+//TODO should include parallel processing of snapshots and document_mode
+
+NOTE: If the pipeline restarts during this process, Redpanda Connect must start the snapshot capture from scratch to store the current oplog position in the <<checkpoint_cache,`checkpoint_cache`>>.
+
+//TODO
+
+Finally, the input uses the stored oplog position to catch up with changes that occurred during snapshot processing.
+
+=== Streaming mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `false`, Redpanda Connect connects to your MongoDB database and starts processing data changes from the latest oplog position. If the pipeline restarts, Redpanda Connect resumes processing updates from the last oplog position written to the <<checkpoint_cache,`checkpoint_cache`>>.
+
+== Data mappings
+
+The following table shows how selected MongoDB data types are mapped to data types supported in Redpanda Connect. All other data types are mapped to string values.
+
+|===
+| MongoDB data type | Bloblang value
+
+| Placeholder
+| Placeholder
+
+|===
+
+== Metadata
+
+This input adds the following metadata fields to each message:
+
+- `operation`: The type of data change that generated the message: `read`, `create`, `update`, `replace`, `delete`, `update`. A `read` operation occurs when the initial snapshot of the database is processed.
+- `collection`: The name of the collection from which the message originated.
+- `operation_time`: The time the data change was written to the https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^].
+
+== Fields
+
+=== `url`
+
+The URL of the target MongoDB server.
+
+*Type*: `string`
+
+*Default*: `""`
+
+```yml
+# Examples
+
+url: mongodb://localhost:27017
+```
+
+=== `database`
+
+The name of the MongoDB database to stream changes from.
+
+*Type*: `string`
+
+*Default*: `""`
+
+
+=== `username`
+
+The username to connect to the database.
+
+*Type*: `string`
+
+*Default*: `""`
+
+=== `password`
+
+The password to connect to the database.
+
+include::components:partial$secret_warning.adoc[]
+
+*Type*: `string`
+
+*Default*: `""`
+
+
+=== `collections`
+
+A list of collections to stream changes from. Specify each collection name as a separate item.
+
+*Type*: `array`
+
+*Default*: `[]`
+
+```yml
+# Examples
+
+collections:
+  - orders
+  - customers
+  - inventory
+```
+
+=== `checkpoint_cache`
+
+Specify a `cache` resource to store the oplog position for the most recent data update streamed to Redpanda Connect. After a restart, Redpanda Connect can continue processing changes from this position, avoiding the need to reprocess all collection updates.
+
+*Type*: `string`
+
+*Default*: `""`
+
+```yml
+# Examples
+
+input:
+  mongodb_cdc:
+    url: mongodb://localhost:27017
+    collections: [my_collection]
+    checkpoint_cache: "my_cdc_cache"
+cache_resources:
+  - label: "my_cdc_cache"
+    redis:
+      url: redis://:6379
+```
+
+=== `checkpoint_key`
+
+The key identifier used to store the oplog position in <<checkpoint_cache,`checkpoint_cache`>>. If you have multiple `mongodb_cdc` inputs sharing the same cache, you can provide an alternative key.
+
+
+*Type*: `string`
+
+*Default*: `mongodb_cdc_checkpoint`
+
+
+=== `checkpoint_interval`
+
+The interval between writing checkpoints to the cache.
+
+*Type*: `string`
+
+*Default*: `5s`
+
+=== `checkpoint_limit`
+
+The maximum number of messages that this input can process at a given time. Increasing this limit enables parallel processing, and batching at the output level. To preserve at-least-once guarantees, any given oplog position is not acknowledged until all messages under that offset are delivered.
+
+*Type*: `int`
+
+*Default*: `1000`
+
+
+=== `read_batch_size`
+
+The number of documents to fetch in each message batch from MongoDB.
path: modules/components/pages/inputs/mongodb_cdc.adoc
user: login: asimms41
body: Will leave as it then.
created_at: 2025-03-07T09:12:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,393 @@
+= mongodb_cdc
+// tag::single-source[]
+:type: input
+:page-beta: true
+:categories: ["Services"]
+
+component_type_dropdown::[]
+
+Streams data changes from a MongoDB replica set, using MongoDB's https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data updates.
+
+ifndef::env-cloud[]
+Introduced in version 4.48.1.
+endif::[]
+
+[tabs]
+======
+Common::
++
+--
+
+```yml
+# Configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: "" # Optional
+    password: "" # Optional
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    auto_replay_nacks: true
+```
+
+--
+Advanced::
++
+--
+
+```yml
+# All configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: ""
+    password: ""
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    snapshot_auto_bucket_sharding: false
+    document_mode: update_lookup
+    json_marshal_mode: canonical
+    app_name: benthos
+    auto_replay_nacks: true
+```
+--
+======
+
+== Prerequisites
+
+- MongoDB version 6 or later
+ifndef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment.
+endif::[]
+ifdef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment. For detailed networking information, including how to set up a VPC peering connection, see xref:networking:index.adoc[Redpanda Cloud Networking].
+endif::[]
+- A sharded MongoDB database or MongoDB cluster running as a replica set
+
+ifndef::env-cloud[]
+== Enable connectivity from cloud-based data sources
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the IP addresses of your Redpanda Connect instances to your firewall rules.
+
+endif::[]
+
+ifdef::env-cloud[]
+
+== Enable connectivity from cloud-based data sources (BYOC)
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the NAT Gateway IP address of your Redpanda cluster to the allowlist of your data source.
+
+endif::[]
+
+== Data capture method
+
+The `mongodb_cdc` input uses https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data changes, which does not propagate _all_ changes to Redpanda Connect. To capture all changes in a MongoDB cluster, including deletions, enable pre- and post-image saving for the cluster and <<database-collections, required collections>>. For more information, see <<document-mode, `document_mode` options>> and the https://www.mongodb.com/docs/manual/changeStreams/#change-streams-with-document-pre--and-post-images[MongoDB documentation].
+
+
+== Data replication
+
+Redpanda Connect allows you to specify which <<collections,database collections>> in your source database to receive changes from.
+
+You can also run the `mongodb_cdc` input in one of two modes, depending on whether you need a snapshot of existing data before streaming updates.
+
+- Snapshot mode: Redpanda Connect first captures a snapshot of all data in the selected collections and streams the contents before processing changes from the last recorded https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^] position.
+- Streaming mode: Redpanda Connect skips the snapshot and processes only the most recent data changes, starting from the latest oplog position.
+
+=== Snapshot mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `true`, Redpanda Connect connects to your MongoDB database and does the following to capture a snapshot of all data in the selected collections:
+
+//TODO should include parallel processing of snapshots and document_mode
+
+NOTE: If the pipeline restarts during this process, Redpanda Connect must start the snapshot capture from scratch to store the current oplog position in the <<checkpoint_cache,`checkpoint_cache`>>.
+
+//TODO
+
+Finally, the input uses the stored oplog position to catch up with changes that occurred during snapshot processing.
+
+=== Streaming mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `false`, Redpanda Connect connects to your MongoDB database and starts processing data changes from the latest oplog position. If the pipeline restarts, Redpanda Connect resumes processing updates from the last oplog position written to the <<checkpoint_cache,`checkpoint_cache`>>.
+
+== Data mappings
path: modules/components/pages/inputs/mongodb_cdc.adoc
user: login: asimms41
body: Removed.
created_at: 2025-03-07T09:14:12Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,393 @@
+= mongodb_cdc
+// tag::single-source[]
+:type: input
+:page-beta: true
+:categories: ["Services"]
+
+component_type_dropdown::[]
+
+Streams data changes from a MongoDB replica set, using MongoDB's https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data updates.
+
+ifndef::env-cloud[]
+Introduced in version 4.48.1.
+endif::[]
+
+[tabs]
+======
+Common::
++
+--
+
+```yml
+# Configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: "" # Optional
+    password: "" # Optional
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    auto_replay_nacks: true
+```
+
+--
+Advanced::
++
+--
+
+```yml
+# All configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: ""
+    password: ""
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    snapshot_auto_bucket_sharding: false
+    document_mode: update_lookup
+    json_marshal_mode: canonical
+    app_name: benthos
+    auto_replay_nacks: true
+```
+--
+======
+
+== Prerequisites
+
+- MongoDB version 6 or later
+ifndef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment.
+endif::[]
+ifdef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment. For detailed networking information, including how to set up a VPC peering connection, see xref:networking:index.adoc[Redpanda Cloud Networking].
+endif::[]
+- A sharded MongoDB database or MongoDB cluster running as a replica set
+
+ifndef::env-cloud[]
+== Enable connectivity from cloud-based data sources
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the IP addresses of your Redpanda Connect instances to your firewall rules.
+
+endif::[]
+
+ifdef::env-cloud[]
+
+== Enable connectivity from cloud-based data sources (BYOC)
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the NAT Gateway IP address of your Redpanda cluster to the allowlist of your data source.
+
+endif::[]
+
+== Data capture method
+
+The `mongodb_cdc` input uses https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data changes, which does not propagate _all_ changes to Redpanda Connect. To capture all changes in a MongoDB cluster, including deletions, enable pre- and post-image saving for the cluster and <<database-collections, required collections>>. For more information, see <<document-mode, `document_mode` options>> and the https://www.mongodb.com/docs/manual/changeStreams/#change-streams-with-document-pre--and-post-images[MongoDB documentation].
+
+
+== Data replication
+
+Redpanda Connect allows you to specify which <<collections,database collections>> in your source database to receive changes from.
+
+You can also run the `mongodb_cdc` input in one of two modes, depending on whether you need a snapshot of existing data before streaming updates.
+
+- Snapshot mode: Redpanda Connect first captures a snapshot of all data in the selected collections and streams the contents before processing changes from the last recorded https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^] position.
+- Streaming mode: Redpanda Connect skips the snapshot and processes only the most recent data changes, starting from the latest oplog position.
+
+=== Snapshot mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `true`, Redpanda Connect connects to your MongoDB database and does the following to capture a snapshot of all data in the selected collections:
+
+//TODO should include parallel processing of snapshots and document_mode
+
+NOTE: If the pipeline restarts during this process, Redpanda Connect must start the snapshot capture from scratch to store the current oplog position in the <<checkpoint_cache,`checkpoint_cache`>>.
+
+//TODO
+
+Finally, the input uses the stored oplog position to catch up with changes that occurred during snapshot processing.
+
+=== Streaming mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `false`, Redpanda Connect connects to your MongoDB database and starts processing data changes from the latest oplog position. If the pipeline restarts, Redpanda Connect resumes processing updates from the last oplog position written to the <<checkpoint_cache,`checkpoint_cache`>>.
+
+== Data mappings
+
+The following table shows how selected MongoDB data types are mapped to data types supported in Redpanda Connect. All other data types are mapped to string values.
+
+|===
+| MongoDB data type | Bloblang value
+
+| Placeholder
+| Placeholder
+
+|===
+
+== Metadata
+
+This input adds the following metadata fields to each message:
+
+- `operation`: The type of data change that generated the message: `read`, `create`, `update`, `replace`, `delete`, `update`. A `read` operation occurs when the initial snapshot of the database is processed.
+- `collection`: The name of the collection from which the message originated.
+- `operation_time`: The time the data change was written to the https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^].
path: modules/components/pages/inputs/mongodb_cdc.adoc
user: login: asimms41
body: Updated.
created_at: 2025-03-07T09:16:21Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,393 @@
+= mongodb_cdc
+// tag::single-source[]
+:type: input
+:page-beta: true
+:categories: ["Services"]
+
+component_type_dropdown::[]
+
+Streams data changes from a MongoDB replica set, using MongoDB's https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data updates.
+
+ifndef::env-cloud[]
+Introduced in version 4.48.1.
+endif::[]
+
+[tabs]
+======
+Common::
++
+--
+
+```yml
+# Configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: "" # Optional
+    password: "" # Optional
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    auto_replay_nacks: true
+```
+
+--
+Advanced::
++
+--
+
+```yml
+# All configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: ""
+    password: ""
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    snapshot_auto_bucket_sharding: false
+    document_mode: update_lookup
+    json_marshal_mode: canonical
+    app_name: benthos
+    auto_replay_nacks: true
+```
+--
+======
+
+== Prerequisites
+
+- MongoDB version 6 or later
+ifndef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment.
+endif::[]
+ifdef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment. For detailed networking information, including how to set up a VPC peering connection, see xref:networking:index.adoc[Redpanda Cloud Networking].
+endif::[]
+- A sharded MongoDB database or MongoDB cluster running as a replica set
+
+ifndef::env-cloud[]
+== Enable connectivity from cloud-based data sources
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the IP addresses of your Redpanda Connect instances to your firewall rules.
+
+endif::[]
+
+ifdef::env-cloud[]
+
+== Enable connectivity from cloud-based data sources (BYOC)
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the NAT Gateway IP address of your Redpanda cluster to the allowlist of your data source.
+
+endif::[]
+
+== Data capture method
+
+The `mongodb_cdc` input uses https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data changes, which does not propagate _all_ changes to Redpanda Connect. To capture all changes in a MongoDB cluster, including deletions, enable pre- and post-image saving for the cluster and <<database-collections, required collections>>. For more information, see <<document-mode, `document_mode` options>> and the https://www.mongodb.com/docs/manual/changeStreams/#change-streams-with-document-pre--and-post-images[MongoDB documentation].
+
+
+== Data replication
+
+Redpanda Connect allows you to specify which <<collections,database collections>> in your source database to receive changes from.
+
+You can also run the `mongodb_cdc` input in one of two modes, depending on whether you need a snapshot of existing data before streaming updates.
+
+- Snapshot mode: Redpanda Connect first captures a snapshot of all data in the selected collections and streams the contents before processing changes from the last recorded https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^] position.
+- Streaming mode: Redpanda Connect skips the snapshot and processes only the most recent data changes, starting from the latest oplog position.
+
+=== Snapshot mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `true`, Redpanda Connect connects to your MongoDB database and does the following to capture a snapshot of all data in the selected collections:
+
+//TODO should include parallel processing of snapshots and document_mode
+
+NOTE: If the pipeline restarts during this process, Redpanda Connect must start the snapshot capture from scratch to store the current oplog position in the <<checkpoint_cache,`checkpoint_cache`>>.
+
+//TODO
+
+Finally, the input uses the stored oplog position to catch up with changes that occurred during snapshot processing.
+
+=== Streaming mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `false`, Redpanda Connect connects to your MongoDB database and starts processing data changes from the latest oplog position. If the pipeline restarts, Redpanda Connect resumes processing updates from the last oplog position written to the <<checkpoint_cache,`checkpoint_cache`>>.
+
+== Data mappings
+
+The following table shows how selected MongoDB data types are mapped to data types supported in Redpanda Connect. All other data types are mapped to string values.
+
+|===
+| MongoDB data type | Bloblang value
+
+| Placeholder
+| Placeholder
+
+|===
+
+== Metadata
+
+This input adds the following metadata fields to each message:
+
+- `operation`: The type of data change that generated the message: `read`, `create`, `update`, `replace`, `delete`, `update`. A `read` operation occurs when the initial snapshot of the database is processed.
+- `collection`: The name of the collection from which the message originated.
+- `operation_time`: The time the data change was written to the https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^].
+
+== Fields
+
+=== `url`
+
+The URL of the target MongoDB server.
+
+*Type*: `string`
+
+*Default*: `""`
+
+```yml
+# Examples
+
+url: mongodb://localhost:27017
+```
+
+=== `database`
+
+The name of the MongoDB database to stream changes from.
+
+*Type*: `string`
+
+*Default*: `""`
+
+
+=== `username`
+
+The username to connect to the database.
+
+*Type*: `string`
+
+*Default*: `""`
+
+=== `password`
+
+The password to connect to the database.
+
+include::components:partial$secret_warning.adoc[]
+
+*Type*: `string`
+
+*Default*: `""`
+
+
+=== `collections`
+
+A list of collections to stream changes from. Specify each collection name as a separate item.
+
+*Type*: `array`
+
+*Default*: `[]`
+
+```yml
+# Examples
+
+collections:
+  - orders
+  - customers
+  - inventory
+```
+
+=== `checkpoint_cache`
+
+Specify a `cache` resource to store the oplog position for the most recent data update streamed to Redpanda Connect. After a restart, Redpanda Connect can continue processing changes from this position, avoiding the need to reprocess all collection updates.
+
+*Type*: `string`
+
+*Default*: `""`
+
+```yml
+# Examples
+
+input:
+  mongodb_cdc:
+    url: mongodb://localhost:27017
+    collections: [my_collection]
+    checkpoint_cache: "my_cdc_cache"
+cache_resources:
+  - label: "my_cdc_cache"
+    redis:
+      url: redis://:6379
+```
+
+=== `checkpoint_key`
+
+The key identifier used to store the oplog position in <<checkpoint_cache,`checkpoint_cache`>>. If you have multiple `mongodb_cdc` inputs sharing the same cache, you can provide an alternative key.
+
+
+*Type*: `string`
+
+*Default*: `mongodb_cdc_checkpoint`
+
+
+=== `checkpoint_interval`
+
+The interval between writing checkpoints to the cache.
+
+*Type*: `string`
+
+*Default*: `5s`
+
+=== `checkpoint_limit`
+
+The maximum number of messages that this input can process at a given time. Increasing this limit enables parallel processing, and batching at the output level. To preserve at-least-once guarantees, any given oplog position is not acknowledged until all messages under that offset are delivered.
+
+*Type*: `int`
+
+*Default*: `1000`
+
+
+=== `read_batch_size`
+
+The number of documents to fetch in each message batch from MongoDB.
+
+
+*Type*: `int`
+
+*Default*: `1000`
+
+
+=== `read_max_wait`
+
+The maximum duration MongoDB waits to accumulate the <<read-batch-size, `read_batch_size`>> documents on a change stream before returning the batch to Redpanda Connect.
+
+*Type*: `string`
+
+*Default*: `1s`
+
+=== `stream_snapshot`
+
+When set to `true`, this input streams a snapshot of all existing data in the source collections before streaming data changes.
+
+*Type*: `bool`
+
+*Default*: `false`
+
+```yml
+# Examples
+
+stream_snapshot: true
+```
+
+=== `snapshot_parallelism`
+
+Specifies the number of connections to use when reading the initial snapshot from one or more collections. Increase this number to enable parallel processing of the snapshot.
+
+This feature uses the `$splitVector` command to split snapshot data into chunks for more efficient processing.
+
+*Type*: `int`
+
+*Default*: `1`
+
+=== `snapshot_auto_bucket_sharding`
+
+Uses the https://www.mongodb.com/docs/manual/reference/operator/aggregation/bucketAuto/[`$bucketAuto`^] command instead of the default, `$splitVector`, to split the snapshot data into chunks for processing. This is required for environments, such as MongoDB Atlas, where privileged access is restricted. To enable parallel processing in these environments:
+
+- Set `snapshot_auto_bucket_sharding` to `true`.
path: modules/components/pages/inputs/mongodb_cdc.adoc
user: login: asimms41
body: I have updated wording to the following so that it's clear that this field needs to be set to true as well:

- Set this field to to `true`.
- Set `stream_snapshot` to `true`.
- Increase `snapshot_parallelism` to a value greater than `1`.
created_at: 2025-03-07T09:24:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,393 @@
+= mongodb_cdc
+// tag::single-source[]
+:type: input
+:page-beta: true
+:categories: ["Services"]
+
+component_type_dropdown::[]
+
+Streams data changes from a MongoDB replica set, using MongoDB's https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data updates.
+
+ifndef::env-cloud[]
+Introduced in version 4.48.1.
+endif::[]
+
+[tabs]
+======
+Common::
++
+--
+
+```yml
+# Configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: "" # Optional
+    password: "" # Optional
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    auto_replay_nacks: true
+```
+
+--
+Advanced::
++
+--
+
+```yml
+# All configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: ""
+    password: ""
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    snapshot_auto_bucket_sharding: false
+    document_mode: update_lookup
+    json_marshal_mode: canonical
+    app_name: benthos
+    auto_replay_nacks: true
+```
+--
+======
+
+== Prerequisites
+
+- MongoDB version 6 or later
+ifndef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment.
+endif::[]
+ifdef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment. For detailed networking information, including how to set up a VPC peering connection, see xref:networking:index.adoc[Redpanda Cloud Networking].
+endif::[]
+- A sharded MongoDB database or MongoDB cluster running as a replica set
+
+ifndef::env-cloud[]
+== Enable connectivity from cloud-based data sources
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the IP addresses of your Redpanda Connect instances to your firewall rules.
+
+endif::[]
+
+ifdef::env-cloud[]
+
+== Enable connectivity from cloud-based data sources (BYOC)
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the NAT Gateway IP address of your Redpanda cluster to the allowlist of your data source.
+
+endif::[]
+
+== Data capture method
+
+The `mongodb_cdc` input uses https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data changes, which does not propagate _all_ changes to Redpanda Connect. To capture all changes in a MongoDB cluster, including deletions, enable pre- and post-image saving for the cluster and <<database-collections, required collections>>. For more information, see <<document-mode, `document_mode` options>> and the https://www.mongodb.com/docs/manual/changeStreams/#change-streams-with-document-pre--and-post-images[MongoDB documentation].
+
+
+== Data replication
+
+Redpanda Connect allows you to specify which <<collections,database collections>> in your source database to receive changes from.
+
+You can also run the `mongodb_cdc` input in one of two modes, depending on whether you need a snapshot of existing data before streaming updates.
+
+- Snapshot mode: Redpanda Connect first captures a snapshot of all data in the selected collections and streams the contents before processing changes from the last recorded https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^] position.
+- Streaming mode: Redpanda Connect skips the snapshot and processes only the most recent data changes, starting from the latest oplog position.
+
+=== Snapshot mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `true`, Redpanda Connect connects to your MongoDB database and does the following to capture a snapshot of all data in the selected collections:
+
+//TODO should include parallel processing of snapshots and document_mode
+
+NOTE: If the pipeline restarts during this process, Redpanda Connect must start the snapshot capture from scratch to store the current oplog position in the <<checkpoint_cache,`checkpoint_cache`>>.
+
+//TODO
+
+Finally, the input uses the stored oplog position to catch up with changes that occurred during snapshot processing.
+
+=== Streaming mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `false`, Redpanda Connect connects to your MongoDB database and starts processing data changes from the latest oplog position. If the pipeline restarts, Redpanda Connect resumes processing updates from the last oplog position written to the <<checkpoint_cache,`checkpoint_cache`>>.
+
+== Data mappings
+
+The following table shows how selected MongoDB data types are mapped to data types supported in Redpanda Connect. All other data types are mapped to string values.
+
+|===
+| MongoDB data type | Bloblang value
+
+| Placeholder
+| Placeholder
+
+|===
+
+== Metadata
+
+This input adds the following metadata fields to each message:
+
+- `operation`: The type of data change that generated the message: `read`, `create`, `update`, `replace`, `delete`, `update`. A `read` operation occurs when the initial snapshot of the database is processed.
+- `collection`: The name of the collection from which the message originated.
+- `operation_time`: The time the data change was written to the https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^].
+
+== Fields
+
+=== `url`
+
+The URL of the target MongoDB server.
+
+*Type*: `string`
+
+*Default*: `""`
+
+```yml
+# Examples
+
+url: mongodb://localhost:27017
+```
+
+=== `database`
+
+The name of the MongoDB database to stream changes from.
+
+*Type*: `string`
+
+*Default*: `""`
+
+
+=== `username`
+
+The username to connect to the database.
+
+*Type*: `string`
+
+*Default*: `""`
+
+=== `password`
+
+The password to connect to the database.
+
+include::components:partial$secret_warning.adoc[]
+
+*Type*: `string`
+
+*Default*: `""`
+
+
+=== `collections`
+
+A list of collections to stream changes from. Specify each collection name as a separate item.
+
+*Type*: `array`
+
+*Default*: `[]`
+
+```yml
+# Examples
+
+collections:
+  - orders
+  - customers
+  - inventory
+```
+
+=== `checkpoint_cache`
+
+Specify a `cache` resource to store the oplog position for the most recent data update streamed to Redpanda Connect. After a restart, Redpanda Connect can continue processing changes from this position, avoiding the need to reprocess all collection updates.
+
+*Type*: `string`
+
+*Default*: `""`
+
+```yml
+# Examples
+
+input:
+  mongodb_cdc:
+    url: mongodb://localhost:27017
+    collections: [my_collection]
+    checkpoint_cache: "my_cdc_cache"
+cache_resources:
+  - label: "my_cdc_cache"
+    redis:
+      url: redis://:6379
+```
+
+=== `checkpoint_key`
+
+The key identifier used to store the oplog position in <<checkpoint_cache,`checkpoint_cache`>>. If you have multiple `mongodb_cdc` inputs sharing the same cache, you can provide an alternative key.
+
+
+*Type*: `string`
+
+*Default*: `mongodb_cdc_checkpoint`
+
+
+=== `checkpoint_interval`
+
+The interval between writing checkpoints to the cache.
+
+*Type*: `string`
+
+*Default*: `5s`
+
+=== `checkpoint_limit`
+
+The maximum number of messages that this input can process at a given time. Increasing this limit enables parallel processing, and batching at the output level. To preserve at-least-once guarantees, any given oplog position is not acknowledged until all messages under that offset are delivered.
+
+*Type*: `int`
+
+*Default*: `1000`
+
+
+=== `read_batch_size`
+
+The number of documents to fetch in each message batch from MongoDB.
+
+
+*Type*: `int`
+
+*Default*: `1000`
+
+
+=== `read_max_wait`
+
+The maximum duration MongoDB waits to accumulate the <<read-batch-size, `read_batch_size`>> documents on a change stream before returning the batch to Redpanda Connect.
+
+*Type*: `string`
+
+*Default*: `1s`
+
+=== `stream_snapshot`
+
+When set to `true`, this input streams a snapshot of all existing data in the source collections before streaming data changes.
+
+*Type*: `bool`
+
+*Default*: `false`
+
+```yml
+# Examples
+
+stream_snapshot: true
+```
+
+=== `snapshot_parallelism`
+
+Specifies the number of connections to use when reading the initial snapshot from one or more collections. Increase this number to enable parallel processing of the snapshot.
+
+This feature uses the `$splitVector` command to split snapshot data into chunks for more efficient processing.
+
+*Type*: `int`
+
+*Default*: `1`
+
+=== `snapshot_auto_bucket_sharding`
+
+Uses the https://www.mongodb.com/docs/manual/reference/operator/aggregation/bucketAuto/[`$bucketAuto`^] command instead of the default, `$splitVector`, to split the snapshot data into chunks for processing. This is required for environments, such as MongoDB Atlas, where privileged access is restricted. To enable parallel processing in these environments:
+
+- Set `snapshot_auto_bucket_sharding` to `true`.
+- Increase `snapshot_parallelism` to a value greater than `1`.
+
+*Type*: `bool`
+
+*Default*: `false`
+
+=== `document_mode`
+
+The mode in which MongoDB emits document changes to Redpanda Connect, specifically updates and deletes.
+
+[cols="1,2"]
+|===
+| Option | Mode description
+
+| `partial_update`
+a| Update operations contain only a description of the changes made to the document, using the following schema:
+
+[,bash,role="no-placeholders"]
+----
+{
+  "_id": "<document_id>",
+  "operations": [
+    # type == set: Indicates that a value was updated. For example:
+    # root.name."address.city" = "New York"
+    {"path": ["name", "address.city"], "type": "set", "value": "New York"},
+
+    # type == unset: Indicates that a value was deleted. For example:
+    # root.phone = deleted()
+    {"path": ["phone"], "type": "unset", "value": null},
+
+    # type == truncatedArray: Indicates that an array was truncated to a specified number of elements. For example:
+    # root.items = this.items.slice(2)
+    {"path": ["items"], "type": "truncatedArray", "value": 2}
+  ]
+}
+----
+
+| `pre_and_post_images`
+a| Emits full documents for update and delete operations by using pre- and post-image collections.  
+For setup and configuration details, see the https://www.mongodb.com/docs/manual/changeStreams/#change-streams-with-document-pre--and-post-images[MongoDB documentation^].
+
+| `update_lookup`
+a| Emits full documents for insert, replace, and update operations, but only the `_id` field for delete operations.  
+This corresponds to the `updateLookup` option.  
+For more information, see the https://www.mongodb.com/docs/manual/changeStreams/#std-label-change-streams-updateLookup[MongoDB documentation^].
+
+|===
+
+=== `json_marshal_mode`
+
+Controls the format of the message output to Redpanda Connect.
+
+*Type*: `string`
+
+*Default*: `canonical`
+
+|===
+| Option | Description
+
+| `canonical`
+| A string format that preserves the exact data types of values stored in MongoDB at the expense of readability and interoperability. Conversion from canonical to Binary JSON (BSON) generally retains type information, except in specific cases.
path: modules/components/pages/inputs/mongodb_cdc.adoc
user: login: asimms41
body: Reading this again, I'm not sure about when the conversion from canonical to BSON is happening. Is this the process?

- MongoDB converts message from canonical to BSON
- RPCN receives the message and converts from BSON to JSON


created_at: 2025-03-07T09:28:15Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,393 @@
+= mongodb_cdc
+// tag::single-source[]
+:type: input
+:page-beta: true
+:categories: ["Services"]
+
+component_type_dropdown::[]
+
+Streams data changes from a MongoDB replica set, using MongoDB's https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data updates.
+
+ifndef::env-cloud[]
+Introduced in version 4.48.1.
+endif::[]
+
+[tabs]
+======
+Common::
++
+--
+
+```yml
+# Configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: "" # Optional
+    password: "" # Optional
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    auto_replay_nacks: true
+```
+
+--
+Advanced::
++
+--
+
+```yml
+# All configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: ""
+    password: ""
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    snapshot_auto_bucket_sharding: false
+    document_mode: update_lookup
+    json_marshal_mode: canonical
+    app_name: benthos
+    auto_replay_nacks: true
+```
+--
+======
+
+== Prerequisites
+
+- MongoDB version 6 or later
+ifndef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment.
+endif::[]
+ifdef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment. For detailed networking information, including how to set up a VPC peering connection, see xref:networking:index.adoc[Redpanda Cloud Networking].
+endif::[]
+- A sharded MongoDB database or MongoDB cluster running as a replica set
+
+ifndef::env-cloud[]
+== Enable connectivity from cloud-based data sources
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the IP addresses of your Redpanda Connect instances to your firewall rules.
+
+endif::[]
+
+ifdef::env-cloud[]
+
+== Enable connectivity from cloud-based data sources (BYOC)
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the NAT Gateway IP address of your Redpanda cluster to the allowlist of your data source.
+
+endif::[]
+
+== Data capture method
+
+The `mongodb_cdc` input uses https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data changes, which does not propagate _all_ changes to Redpanda Connect. To capture all changes in a MongoDB cluster, including deletions, enable pre- and post-image saving for the cluster and <<database-collections, required collections>>. For more information, see <<document-mode, `document_mode` options>> and the https://www.mongodb.com/docs/manual/changeStreams/#change-streams-with-document-pre--and-post-images[MongoDB documentation].
+
+
+== Data replication
+
+Redpanda Connect allows you to specify which <<collections,database collections>> in your source database to receive changes from.
+
+You can also run the `mongodb_cdc` input in one of two modes, depending on whether you need a snapshot of existing data before streaming updates.
+
+- Snapshot mode: Redpanda Connect first captures a snapshot of all data in the selected collections and streams the contents before processing changes from the last recorded https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^] position.
+- Streaming mode: Redpanda Connect skips the snapshot and processes only the most recent data changes, starting from the latest oplog position.
+
+=== Snapshot mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `true`, Redpanda Connect connects to your MongoDB database and does the following to capture a snapshot of all data in the selected collections:
+
+//TODO should include parallel processing of snapshots and document_mode
+
+NOTE: If the pipeline restarts during this process, Redpanda Connect must start the snapshot capture from scratch to store the current oplog position in the <<checkpoint_cache,`checkpoint_cache`>>.
+
+//TODO
+
+Finally, the input uses the stored oplog position to catch up with changes that occurred during snapshot processing.
+
+=== Streaming mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `false`, Redpanda Connect connects to your MongoDB database and starts processing data changes from the latest oplog position. If the pipeline restarts, Redpanda Connect resumes processing updates from the last oplog position written to the <<checkpoint_cache,`checkpoint_cache`>>.
+
+== Data mappings
+
+The following table shows how selected MongoDB data types are mapped to data types supported in Redpanda Connect. All other data types are mapped to string values.
+
+|===
+| MongoDB data type | Bloblang value
+
+| Placeholder
+| Placeholder
+
+|===
+
+== Metadata
+
+This input adds the following metadata fields to each message:
+
+- `operation`: The type of data change that generated the message: `read`, `create`, `update`, `replace`, `delete`, `update`. A `read` operation occurs when the initial snapshot of the database is processed.
+- `collection`: The name of the collection from which the message originated.
+- `operation_time`: The time the data change was written to the https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^].
+
+== Fields
+
+=== `url`
+
+The URL of the target MongoDB server.
+
+*Type*: `string`
+
+*Default*: `""`
+
+```yml
+# Examples
+
+url: mongodb://localhost:27017
+```
+
+=== `database`
+
+The name of the MongoDB database to stream changes from.
+
+*Type*: `string`
+
+*Default*: `""`
+
+
+=== `username`
+
+The username to connect to the database.
+
+*Type*: `string`
+
+*Default*: `""`
+
+=== `password`
+
+The password to connect to the database.
+
+include::components:partial$secret_warning.adoc[]
+
+*Type*: `string`
+
+*Default*: `""`
+
+
+=== `collections`
+
+A list of collections to stream changes from. Specify each collection name as a separate item.
+
+*Type*: `array`
+
+*Default*: `[]`
+
+```yml
+# Examples
+
+collections:
+  - orders
+  - customers
+  - inventory
+```
+
+=== `checkpoint_cache`
+
+Specify a `cache` resource to store the oplog position for the most recent data update streamed to Redpanda Connect. After a restart, Redpanda Connect can continue processing changes from this position, avoiding the need to reprocess all collection updates.
+
+*Type*: `string`
+
+*Default*: `""`
+
+```yml
+# Examples
+
+input:
+  mongodb_cdc:
+    url: mongodb://localhost:27017
+    collections: [my_collection]
+    checkpoint_cache: "my_cdc_cache"
+cache_resources:
+  - label: "my_cdc_cache"
+    redis:
+      url: redis://:6379
+```
+
+=== `checkpoint_key`
+
+The key identifier used to store the oplog position in <<checkpoint_cache,`checkpoint_cache`>>. If you have multiple `mongodb_cdc` inputs sharing the same cache, you can provide an alternative key.
+
+
+*Type*: `string`
+
+*Default*: `mongodb_cdc_checkpoint`
+
+
+=== `checkpoint_interval`
+
+The interval between writing checkpoints to the cache.
+
+*Type*: `string`
+
+*Default*: `5s`
+
+=== `checkpoint_limit`
+
+The maximum number of messages that this input can process at a given time. Increasing this limit enables parallel processing, and batching at the output level. To preserve at-least-once guarantees, any given oplog position is not acknowledged until all messages under that offset are delivered.
+
+*Type*: `int`
+
+*Default*: `1000`
+
+
+=== `read_batch_size`
+
+The number of documents to fetch in each message batch from MongoDB.
+
+
+*Type*: `int`
+
+*Default*: `1000`
+
+
+=== `read_max_wait`
+
+The maximum duration MongoDB waits to accumulate the <<read-batch-size, `read_batch_size`>> documents on a change stream before returning the batch to Redpanda Connect.
+
+*Type*: `string`
+
+*Default*: `1s`
+
+=== `stream_snapshot`
+
+When set to `true`, this input streams a snapshot of all existing data in the source collections before streaming data changes.
+
+*Type*: `bool`
+
+*Default*: `false`
+
+```yml
+# Examples
+
+stream_snapshot: true
+```
+
+=== `snapshot_parallelism`
+
+Specifies the number of connections to use when reading the initial snapshot from one or more collections. Increase this number to enable parallel processing of the snapshot.
+
+This feature uses the `$splitVector` command to split snapshot data into chunks for more efficient processing.
+
+*Type*: `int`
+
+*Default*: `1`
+
+=== `snapshot_auto_bucket_sharding`
+
+Uses the https://www.mongodb.com/docs/manual/reference/operator/aggregation/bucketAuto/[`$bucketAuto`^] command instead of the default, `$splitVector`, to split the snapshot data into chunks for processing. This is required for environments, such as MongoDB Atlas, where privileged access is restricted. To enable parallel processing in these environments:
+
+- Set `snapshot_auto_bucket_sharding` to `true`.
+- Increase `snapshot_parallelism` to a value greater than `1`.
+
+*Type*: `bool`
+
+*Default*: `false`
+
+=== `document_mode`
+
+The mode in which MongoDB emits document changes to Redpanda Connect, specifically updates and deletes.
+
+[cols="1,2"]
+|===
+| Option | Mode description
+
+| `partial_update`
+a| Update operations contain only a description of the changes made to the document, using the following schema:
+
+[,bash,role="no-placeholders"]
+----
+{
+  "_id": "<document_id>",
+  "operations": [
+    # type == set: Indicates that a value was updated. For example:
+    # root.name."address.city" = "New York"
+    {"path": ["name", "address.city"], "type": "set", "value": "New York"},
+
+    # type == unset: Indicates that a value was deleted. For example:
+    # root.phone = deleted()
+    {"path": ["phone"], "type": "unset", "value": null},
+
+    # type == truncatedArray: Indicates that an array was truncated to a specified number of elements. For example:
+    # root.items = this.items.slice(2)
+    {"path": ["items"], "type": "truncatedArray", "value": 2}
+  ]
+}
+----
+
+| `pre_and_post_images`
+a| Emits full documents for update and delete operations by using pre- and post-image collections.  
+For setup and configuration details, see the https://www.mongodb.com/docs/manual/changeStreams/#change-streams-with-document-pre--and-post-images[MongoDB documentation^].
+
+| `update_lookup`
+a| Emits full documents for insert, replace, and update operations, but only the `_id` field for delete operations.  
+This corresponds to the `updateLookup` option.  
+For more information, see the https://www.mongodb.com/docs/manual/changeStreams/#std-label-change-streams-updateLookup[MongoDB documentation^].
+
+|===
+
+=== `json_marshal_mode`
+
+Controls the format of the message output to Redpanda Connect.
+
+*Type*: `string`
+
+*Default*: `canonical`
+
+|===
+| Option | Description
+
+| `canonical`
+| A string format that preserves the exact data types of values stored in MongoDB at the expense of readability and interoperability. Conversion from canonical to Binary JSON (BSON) generally retains type information, except in specific cases.
path: modules/components/pages/inputs/mongodb_cdc.adoc
user: login: rockwotj
body: Mongo sends us bson. We convert that to json in the format based on this option
created_at: 2025-03-07T09:29:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,393 @@
+= mongodb_cdc
+// tag::single-source[]
+:type: input
+:page-beta: true
+:categories: ["Services"]
+
+component_type_dropdown::[]
+
+Streams data changes from a MongoDB replica set, using MongoDB's https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data updates.
+
+ifndef::env-cloud[]
+Introduced in version 4.48.1.
+endif::[]
+
+[tabs]
+======
+Common::
++
+--
+
+```yml
+# Configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: "" # Optional
+    password: "" # Optional
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    auto_replay_nacks: true
+```
+
+--
+Advanced::
++
+--
+
+```yml
+# All configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: ""
+    password: ""
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    snapshot_auto_bucket_sharding: false
+    document_mode: update_lookup
+    json_marshal_mode: canonical
+    app_name: benthos
+    auto_replay_nacks: true
+```
+--
+======
+
+== Prerequisites
+
+- MongoDB version 6 or later
+ifndef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment.
+endif::[]
+ifdef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment. For detailed networking information, including how to set up a VPC peering connection, see xref:networking:index.adoc[Redpanda Cloud Networking].
+endif::[]
+- A sharded MongoDB database or MongoDB cluster running as a replica set
+
+ifndef::env-cloud[]
+== Enable connectivity from cloud-based data sources
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the IP addresses of your Redpanda Connect instances to your firewall rules.
+
+endif::[]
+
+ifdef::env-cloud[]
+
+== Enable connectivity from cloud-based data sources (BYOC)
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the NAT Gateway IP address of your Redpanda cluster to the allowlist of your data source.
+
+endif::[]
+
+== Data capture method
+
+The `mongodb_cdc` input uses https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data changes, which does not propagate _all_ changes to Redpanda Connect. To capture all changes in a MongoDB cluster, including deletions, enable pre- and post-image saving for the cluster and <<database-collections, required collections>>. For more information, see <<document-mode, `document_mode` options>> and the https://www.mongodb.com/docs/manual/changeStreams/#change-streams-with-document-pre--and-post-images[MongoDB documentation].
+
+
+== Data replication
+
+Redpanda Connect allows you to specify which <<collections,database collections>> in your source database to receive changes from.
+
+You can also run the `mongodb_cdc` input in one of two modes, depending on whether you need a snapshot of existing data before streaming updates.
+
+- Snapshot mode: Redpanda Connect first captures a snapshot of all data in the selected collections and streams the contents before processing changes from the last recorded https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^] position.
+- Streaming mode: Redpanda Connect skips the snapshot and processes only the most recent data changes, starting from the latest oplog position.
+
+=== Snapshot mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `true`, Redpanda Connect connects to your MongoDB database and does the following to capture a snapshot of all data in the selected collections:
+
+//TODO should include parallel processing of snapshots and document_mode
+
+NOTE: If the pipeline restarts during this process, Redpanda Connect must start the snapshot capture from scratch to store the current oplog position in the <<checkpoint_cache,`checkpoint_cache`>>.
+
+//TODO
+
+Finally, the input uses the stored oplog position to catch up with changes that occurred during snapshot processing.
+
+=== Streaming mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `false`, Redpanda Connect connects to your MongoDB database and starts processing data changes from the latest oplog position. If the pipeline restarts, Redpanda Connect resumes processing updates from the last oplog position written to the <<checkpoint_cache,`checkpoint_cache`>>.
+
+== Data mappings
+
+The following table shows how selected MongoDB data types are mapped to data types supported in Redpanda Connect. All other data types are mapped to string values.
+
+|===
+| MongoDB data type | Bloblang value
+
+| Placeholder
+| Placeholder
+
+|===
+
+== Metadata
+
+This input adds the following metadata fields to each message:
+
+- `operation`: The type of data change that generated the message: `read`, `create`, `update`, `replace`, `delete`, `update`. A `read` operation occurs when the initial snapshot of the database is processed.
+- `collection`: The name of the collection from which the message originated.
+- `operation_time`: The time the data change was written to the https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^].
+
+== Fields
+
+=== `url`
+
+The URL of the target MongoDB server.
+
+*Type*: `string`
+
+*Default*: `""`
+
+```yml
+# Examples
+
+url: mongodb://localhost:27017
+```
+
+=== `database`
+
+The name of the MongoDB database to stream changes from.
+
+*Type*: `string`
+
+*Default*: `""`
+
+
+=== `username`
+
+The username to connect to the database.
+
+*Type*: `string`
+
+*Default*: `""`
+
+=== `password`
+
+The password to connect to the database.
+
+include::components:partial$secret_warning.adoc[]
+
+*Type*: `string`
+
+*Default*: `""`
+
+
+=== `collections`
+
+A list of collections to stream changes from. Specify each collection name as a separate item.
+
+*Type*: `array`
+
+*Default*: `[]`
+
+```yml
+# Examples
+
+collections:
+  - orders
+  - customers
+  - inventory
+```
+
+=== `checkpoint_cache`
+
+Specify a `cache` resource to store the oplog position for the most recent data update streamed to Redpanda Connect. After a restart, Redpanda Connect can continue processing changes from this position, avoiding the need to reprocess all collection updates.
+
+*Type*: `string`
+
+*Default*: `""`
+
+```yml
+# Examples
+
+input:
+  mongodb_cdc:
+    url: mongodb://localhost:27017
+    collections: [my_collection]
+    checkpoint_cache: "my_cdc_cache"
+cache_resources:
+  - label: "my_cdc_cache"
+    redis:
+      url: redis://:6379
+```
+
+=== `checkpoint_key`
+
+The key identifier used to store the oplog position in <<checkpoint_cache,`checkpoint_cache`>>. If you have multiple `mongodb_cdc` inputs sharing the same cache, you can provide an alternative key.
+
+
+*Type*: `string`
+
+*Default*: `mongodb_cdc_checkpoint`
+
+
+=== `checkpoint_interval`
+
+The interval between writing checkpoints to the cache.
+
+*Type*: `string`
+
+*Default*: `5s`
+
+=== `checkpoint_limit`
+
+The maximum number of messages that this input can process at a given time. Increasing this limit enables parallel processing, and batching at the output level. To preserve at-least-once guarantees, any given oplog position is not acknowledged until all messages under that offset are delivered.
+
+*Type*: `int`
+
+*Default*: `1000`
+
+
+=== `read_batch_size`
+
+The number of documents to fetch in each message batch from MongoDB.
+
+
+*Type*: `int`
+
+*Default*: `1000`
+
+
+=== `read_max_wait`
+
+The maximum duration MongoDB waits to accumulate the <<read-batch-size, `read_batch_size`>> documents on a change stream before returning the batch to Redpanda Connect.
+
+*Type*: `string`
+
+*Default*: `1s`
+
+=== `stream_snapshot`
+
+When set to `true`, this input streams a snapshot of all existing data in the source collections before streaming data changes.
+
+*Type*: `bool`
+
+*Default*: `false`
+
+```yml
+# Examples
+
+stream_snapshot: true
+```
+
+=== `snapshot_parallelism`
+
+Specifies the number of connections to use when reading the initial snapshot from one or more collections. Increase this number to enable parallel processing of the snapshot.
+
+This feature uses the `$splitVector` command to split snapshot data into chunks for more efficient processing.
+
+*Type*: `int`
+
+*Default*: `1`
+
+=== `snapshot_auto_bucket_sharding`
+
+Uses the https://www.mongodb.com/docs/manual/reference/operator/aggregation/bucketAuto/[`$bucketAuto`^] command instead of the default, `$splitVector`, to split the snapshot data into chunks for processing. This is required for environments, such as MongoDB Atlas, where privileged access is restricted. To enable parallel processing in these environments:
+
+- Set `snapshot_auto_bucket_sharding` to `true`.
+- Increase `snapshot_parallelism` to a value greater than `1`.
+
+*Type*: `bool`
+
+*Default*: `false`
+
+=== `document_mode`
+
+The mode in which MongoDB emits document changes to Redpanda Connect, specifically updates and deletes.
+
+[cols="1,2"]
+|===
+| Option | Mode description
+
+| `partial_update`
+a| Update operations contain only a description of the changes made to the document, using the following schema:
+
+[,bash,role="no-placeholders"]
+----
+{
+  "_id": "<document_id>",
+  "operations": [
+    # type == set: Indicates that a value was updated. For example:
+    # root.name."address.city" = "New York"
+    {"path": ["name", "address.city"], "type": "set", "value": "New York"},
+
+    # type == unset: Indicates that a value was deleted. For example:
+    # root.phone = deleted()
+    {"path": ["phone"], "type": "unset", "value": null},
+
+    # type == truncatedArray: Indicates that an array was truncated to a specified number of elements. For example:
+    # root.items = this.items.slice(2)
+    {"path": ["items"], "type": "truncatedArray", "value": 2}
+  ]
+}
+----
+
+| `pre_and_post_images`
+a| Emits full documents for update and delete operations by using pre- and post-image collections.  
+For setup and configuration details, see the https://www.mongodb.com/docs/manual/changeStreams/#change-streams-with-document-pre--and-post-images[MongoDB documentation^].
+
+| `update_lookup`
+a| Emits full documents for insert, replace, and update operations, but only the `_id` field for delete operations.  
+This corresponds to the `updateLookup` option.  
+For more information, see the https://www.mongodb.com/docs/manual/changeStreams/#std-label-change-streams-updateLookup[MongoDB documentation^].
+
+|===
+
+=== `json_marshal_mode`
+
+Controls the format of the message output to Redpanda Connect.
+
+*Type*: `string`
+
+*Default*: `canonical`
+
+|===
+| Option | Description
+
+| `canonical`
+| A string format that preserves the exact data types of values stored in MongoDB at the expense of readability and interoperability. Conversion from canonical to Binary JSON (BSON) generally retains type information, except in specific cases.
path: modules/components/pages/inputs/mongodb_cdc.adoc
user: login: rockwotj
body: Also not sure if it's helpful but here is the spec for canonical vs relaxed json: https://github.com/mongodb/specifications/blob/master/source/extended-json/extended-json.md#extended-json-format
created_at: 2025-03-07T09:31:42Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,393 @@
+= mongodb_cdc
+// tag::single-source[]
+:type: input
+:page-beta: true
+:categories: ["Services"]
+
+component_type_dropdown::[]
+
+Streams data changes from a MongoDB replica set, using MongoDB's https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data updates.
+
+ifndef::env-cloud[]
+Introduced in version 4.48.1.
+endif::[]
+
+[tabs]
+======
+Common::
++
+--
+
+```yml
+# Configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: "" # Optional
+    password: "" # Optional
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    auto_replay_nacks: true
+```
+
+--
+Advanced::
++
+--
+
+```yml
+# All configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: ""
+    password: ""
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    snapshot_auto_bucket_sharding: false
+    document_mode: update_lookup
+    json_marshal_mode: canonical
+    app_name: benthos
+    auto_replay_nacks: true
+```
+--
+======
+
+== Prerequisites
+
+- MongoDB version 6 or later
+ifndef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment.
+endif::[]
+ifdef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment. For detailed networking information, including how to set up a VPC peering connection, see xref:networking:index.adoc[Redpanda Cloud Networking].
+endif::[]
+- A sharded MongoDB database or MongoDB cluster running as a replica set
+
+ifndef::env-cloud[]
+== Enable connectivity from cloud-based data sources
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the IP addresses of your Redpanda Connect instances to your firewall rules.
+
+endif::[]
+
+ifdef::env-cloud[]
+
+== Enable connectivity from cloud-based data sources (BYOC)
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the NAT Gateway IP address of your Redpanda cluster to the allowlist of your data source.
+
+endif::[]
+
+== Data capture method
+
+The `mongodb_cdc` input uses https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data changes, which does not propagate _all_ changes to Redpanda Connect. To capture all changes in a MongoDB cluster, including deletions, enable pre- and post-image saving for the cluster and <<database-collections, required collections>>. For more information, see <<document-mode, `document_mode` options>> and the https://www.mongodb.com/docs/manual/changeStreams/#change-streams-with-document-pre--and-post-images[MongoDB documentation].
+
+
+== Data replication
+
+Redpanda Connect allows you to specify which <<collections,database collections>> in your source database to receive changes from.
+
+You can also run the `mongodb_cdc` input in one of two modes, depending on whether you need a snapshot of existing data before streaming updates.
+
+- Snapshot mode: Redpanda Connect first captures a snapshot of all data in the selected collections and streams the contents before processing changes from the last recorded https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^] position.
+- Streaming mode: Redpanda Connect skips the snapshot and processes only the most recent data changes, starting from the latest oplog position.
+
+=== Snapshot mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `true`, Redpanda Connect connects to your MongoDB database and does the following to capture a snapshot of all data in the selected collections:
+
+//TODO should include parallel processing of snapshots and document_mode
+
+NOTE: If the pipeline restarts during this process, Redpanda Connect must start the snapshot capture from scratch to store the current oplog position in the <<checkpoint_cache,`checkpoint_cache`>>.
+
+//TODO
+
+Finally, the input uses the stored oplog position to catch up with changes that occurred during snapshot processing.
+
+=== Streaming mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `false`, Redpanda Connect connects to your MongoDB database and starts processing data changes from the latest oplog position. If the pipeline restarts, Redpanda Connect resumes processing updates from the last oplog position written to the <<checkpoint_cache,`checkpoint_cache`>>.
+
+== Data mappings
+
+The following table shows how selected MongoDB data types are mapped to data types supported in Redpanda Connect. All other data types are mapped to string values.
+
+|===
+| MongoDB data type | Bloblang value
+
+| Placeholder
+| Placeholder
+
+|===
+
+== Metadata
+
+This input adds the following metadata fields to each message:
+
+- `operation`: The type of data change that generated the message: `read`, `create`, `update`, `replace`, `delete`, `update`. A `read` operation occurs when the initial snapshot of the database is processed.
+- `collection`: The name of the collection from which the message originated.
+- `operation_time`: The time the data change was written to the https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^].
+
+== Fields
+
+=== `url`
+
+The URL of the target MongoDB server.
+
+*Type*: `string`
+
+*Default*: `""`
+
+```yml
+# Examples
+
+url: mongodb://localhost:27017
+```
+
+=== `database`
+
+The name of the MongoDB database to stream changes from.
+
+*Type*: `string`
+
+*Default*: `""`
+
+
+=== `username`
+
+The username to connect to the database.
+
+*Type*: `string`
+
+*Default*: `""`
+
+=== `password`
+
+The password to connect to the database.
+
+include::components:partial$secret_warning.adoc[]
+
+*Type*: `string`
+
+*Default*: `""`
+
+
+=== `collections`
+
+A list of collections to stream changes from. Specify each collection name as a separate item.
+
+*Type*: `array`
+
+*Default*: `[]`
+
+```yml
+# Examples
+
+collections:
+  - orders
+  - customers
+  - inventory
+```
+
+=== `checkpoint_cache`
+
+Specify a `cache` resource to store the oplog position for the most recent data update streamed to Redpanda Connect. After a restart, Redpanda Connect can continue processing changes from this position, avoiding the need to reprocess all collection updates.
+
+*Type*: `string`
+
+*Default*: `""`
+
+```yml
+# Examples
+
+input:
+  mongodb_cdc:
+    url: mongodb://localhost:27017
+    collections: [my_collection]
+    checkpoint_cache: "my_cdc_cache"
+cache_resources:
+  - label: "my_cdc_cache"
+    redis:
+      url: redis://:6379
+```
+
+=== `checkpoint_key`
+
+The key identifier used to store the oplog position in <<checkpoint_cache,`checkpoint_cache`>>. If you have multiple `mongodb_cdc` inputs sharing the same cache, you can provide an alternative key.
+
+
+*Type*: `string`
+
+*Default*: `mongodb_cdc_checkpoint`
+
+
+=== `checkpoint_interval`
+
+The interval between writing checkpoints to the cache.
+
+*Type*: `string`
+
+*Default*: `5s`
+
+=== `checkpoint_limit`
+
+The maximum number of messages that this input can process at a given time. Increasing this limit enables parallel processing, and batching at the output level. To preserve at-least-once guarantees, any given oplog position is not acknowledged until all messages under that offset are delivered.
+
+*Type*: `int`
+
+*Default*: `1000`
+
+
+=== `read_batch_size`
+
+The number of documents to fetch in each message batch from MongoDB.
+
+
+*Type*: `int`
+
+*Default*: `1000`
+
+
+=== `read_max_wait`
+
+The maximum duration MongoDB waits to accumulate the <<read-batch-size, `read_batch_size`>> documents on a change stream before returning the batch to Redpanda Connect.
+
+*Type*: `string`
+
+*Default*: `1s`
+
+=== `stream_snapshot`
+
+When set to `true`, this input streams a snapshot of all existing data in the source collections before streaming data changes.
+
+*Type*: `bool`
+
+*Default*: `false`
+
+```yml
+# Examples
+
+stream_snapshot: true
+```
+
+=== `snapshot_parallelism`
+
+Specifies the number of connections to use when reading the initial snapshot from one or more collections. Increase this number to enable parallel processing of the snapshot.
+
+This feature uses the `$splitVector` command to split snapshot data into chunks for more efficient processing.
+
+*Type*: `int`
+
+*Default*: `1`
+
+=== `snapshot_auto_bucket_sharding`
+
+Uses the https://www.mongodb.com/docs/manual/reference/operator/aggregation/bucketAuto/[`$bucketAuto`^] command instead of the default, `$splitVector`, to split the snapshot data into chunks for processing. This is required for environments, such as MongoDB Atlas, where privileged access is restricted. To enable parallel processing in these environments:
+
+- Set `snapshot_auto_bucket_sharding` to `true`.
+- Increase `snapshot_parallelism` to a value greater than `1`.
+
+*Type*: `bool`
+
+*Default*: `false`
+
+=== `document_mode`
+
+The mode in which MongoDB emits document changes to Redpanda Connect, specifically updates and deletes.
+
+[cols="1,2"]
+|===
+| Option | Mode description
+
+| `partial_update`
+a| Update operations contain only a description of the changes made to the document, using the following schema:
+
+[,bash,role="no-placeholders"]
+----
+{
+  "_id": "<document_id>",
+  "operations": [
+    # type == set: Indicates that a value was updated. For example:
+    # root.name."address.city" = "New York"
+    {"path": ["name", "address.city"], "type": "set", "value": "New York"},
+
+    # type == unset: Indicates that a value was deleted. For example:
+    # root.phone = deleted()
+    {"path": ["phone"], "type": "unset", "value": null},
+
+    # type == truncatedArray: Indicates that an array was truncated to a specified number of elements. For example:
+    # root.items = this.items.slice(2)
+    {"path": ["items"], "type": "truncatedArray", "value": 2}
+  ]
+}
+----
+
+| `pre_and_post_images`
+a| Emits full documents for update and delete operations by using pre- and post-image collections.  
+For setup and configuration details, see the https://www.mongodb.com/docs/manual/changeStreams/#change-streams-with-document-pre--and-post-images[MongoDB documentation^].
+
+| `update_lookup`
+a| Emits full documents for insert, replace, and update operations, but only the `_id` field for delete operations.  
+This corresponds to the `updateLookup` option.  
+For more information, see the https://www.mongodb.com/docs/manual/changeStreams/#std-label-change-streams-updateLookup[MongoDB documentation^].
+
+|===
+
+=== `json_marshal_mode`
+
+Controls the format of the message output to Redpanda Connect.
+
+*Type*: `string`
+
+*Default*: `canonical`
+
+|===
+| Option | Description
+
+| `canonical`
+| A string format that preserves the exact data types of values stored in MongoDB at the expense of readability and interoperability. Conversion from canonical to Binary JSON (BSON) generally retains type information, except in specific cases.
path: modules/components/pages/inputs/mongodb_cdc.adoc
user: login: rockwotj
body: Or maybe their docs are better: https://www.mongodb.com/docs/manual/reference/mongodb-extended-json/
created_at: 2025-03-07T09:33:02Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,393 @@
+= mongodb_cdc
+// tag::single-source[]
+:type: input
+:page-beta: true
+:categories: ["Services"]
+
+component_type_dropdown::[]
+
+Streams data changes from a MongoDB replica set, using MongoDB's https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data updates.
+
+ifndef::env-cloud[]
+Introduced in version 4.48.1.
+endif::[]
+
+[tabs]
+======
+Common::
++
+--
+
+```yml
+# Configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: "" # Optional
+    password: "" # Optional
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    auto_replay_nacks: true
+```
+
+--
+Advanced::
++
+--
+
+```yml
+# All configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: ""
+    password: ""
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    snapshot_auto_bucket_sharding: false
+    document_mode: update_lookup
+    json_marshal_mode: canonical
+    app_name: benthos
+    auto_replay_nacks: true
+```
+--
+======
+
+== Prerequisites
+
+- MongoDB version 6 or later
+ifndef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment.
+endif::[]
+ifdef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment. For detailed networking information, including how to set up a VPC peering connection, see xref:networking:index.adoc[Redpanda Cloud Networking].
+endif::[]
+- A sharded MongoDB database or MongoDB cluster running as a replica set
+
+ifndef::env-cloud[]
+== Enable connectivity from cloud-based data sources
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the IP addresses of your Redpanda Connect instances to your firewall rules.
+
+endif::[]
+
+ifdef::env-cloud[]
+
+== Enable connectivity from cloud-based data sources (BYOC)
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the NAT Gateway IP address of your Redpanda cluster to the allowlist of your data source.
+
+endif::[]
+
+== Data capture method
+
+The `mongodb_cdc` input uses https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data changes, which does not propagate _all_ changes to Redpanda Connect. To capture all changes in a MongoDB cluster, including deletions, enable pre- and post-image saving for the cluster and <<database-collections, required collections>>. For more information, see <<document-mode, `document_mode` options>> and the https://www.mongodb.com/docs/manual/changeStreams/#change-streams-with-document-pre--and-post-images[MongoDB documentation].
+
+
+== Data replication
+
+Redpanda Connect allows you to specify which <<collections,database collections>> in your source database to receive changes from.
+
+You can also run the `mongodb_cdc` input in one of two modes, depending on whether you need a snapshot of existing data before streaming updates.
+
+- Snapshot mode: Redpanda Connect first captures a snapshot of all data in the selected collections and streams the contents before processing changes from the last recorded https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^] position.
+- Streaming mode: Redpanda Connect skips the snapshot and processes only the most recent data changes, starting from the latest oplog position.
+
+=== Snapshot mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `true`, Redpanda Connect connects to your MongoDB database and does the following to capture a snapshot of all data in the selected collections:
+
+//TODO should include parallel processing of snapshots and document_mode
path: modules/components/pages/inputs/mongodb_cdc.adoc
user: login: asimms41
body: Updated, hopefully the phrasing is OK.
created_at: 2025-03-07T09:57:56Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,393 @@
+= mongodb_cdc
+// tag::single-source[]
+:type: input
+:page-beta: true
+:categories: ["Services"]
+
+component_type_dropdown::[]
+
+Streams data changes from a MongoDB replica set, using MongoDB's https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data updates.
+
+ifndef::env-cloud[]
+Introduced in version 4.48.1.
+endif::[]
+
+[tabs]
+======
+Common::
++
+--
+
+```yml
+# Configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: "" # Optional
+    password: "" # Optional
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    auto_replay_nacks: true
+```
+
+--
+Advanced::
++
+--
+
+```yml
+# All configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: ""
+    password: ""
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    snapshot_auto_bucket_sharding: false
+    document_mode: update_lookup
+    json_marshal_mode: canonical
+    app_name: benthos
+    auto_replay_nacks: true
+```
+--
+======
+
+== Prerequisites
+
+- MongoDB version 6 or later
+ifndef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment.
+endif::[]
+ifdef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment. For detailed networking information, including how to set up a VPC peering connection, see xref:networking:index.adoc[Redpanda Cloud Networking].
+endif::[]
+- A sharded MongoDB database or MongoDB cluster running as a replica set
+
+ifndef::env-cloud[]
+== Enable connectivity from cloud-based data sources
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the IP addresses of your Redpanda Connect instances to your firewall rules.
+
+endif::[]
+
+ifdef::env-cloud[]
+
+== Enable connectivity from cloud-based data sources (BYOC)
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the NAT Gateway IP address of your Redpanda cluster to the allowlist of your data source.
+
+endif::[]
+
+== Data capture method
+
+The `mongodb_cdc` input uses https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data changes, which does not propagate _all_ changes to Redpanda Connect. To capture all changes in a MongoDB cluster, including deletions, enable pre- and post-image saving for the cluster and <<database-collections, required collections>>. For more information, see <<document-mode, `document_mode` options>> and the https://www.mongodb.com/docs/manual/changeStreams/#change-streams-with-document-pre--and-post-images[MongoDB documentation].
+
+
+== Data replication
+
+Redpanda Connect allows you to specify which <<collections,database collections>> in your source database to receive changes from.
+
+You can also run the `mongodb_cdc` input in one of two modes, depending on whether you need a snapshot of existing data before streaming updates.
+
+- Snapshot mode: Redpanda Connect first captures a snapshot of all data in the selected collections and streams the contents before processing changes from the last recorded https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^] position.
+- Streaming mode: Redpanda Connect skips the snapshot and processes only the most recent data changes, starting from the latest oplog position.
+
+=== Snapshot mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `true`, Redpanda Connect connects to your MongoDB database and does the following to capture a snapshot of all data in the selected collections:
+
+//TODO should include parallel processing of snapshots and document_mode
+
+NOTE: If the pipeline restarts during this process, Redpanda Connect must start the snapshot capture from scratch to store the current oplog position in the <<checkpoint_cache,`checkpoint_cache`>>.
+
+//TODO
+
+Finally, the input uses the stored oplog position to catch up with changes that occurred during snapshot processing.
+
+=== Streaming mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `false`, Redpanda Connect connects to your MongoDB database and starts processing data changes from the latest oplog position. If the pipeline restarts, Redpanda Connect resumes processing updates from the last oplog position written to the <<checkpoint_cache,`checkpoint_cache`>>.
+
+== Data mappings
+
+The following table shows how selected MongoDB data types are mapped to data types supported in Redpanda Connect. All other data types are mapped to string values.
+
+|===
+| MongoDB data type | Bloblang value
+
+| Placeholder
+| Placeholder
+
+|===
+
+== Metadata
+
+This input adds the following metadata fields to each message:
+
+- `operation`: The type of data change that generated the message: `read`, `create`, `update`, `replace`, `delete`, `update`. A `read` operation occurs when the initial snapshot of the database is processed.
+- `collection`: The name of the collection from which the message originated.
+- `operation_time`: The time the data change was written to the https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^].
+
+== Fields
+
+=== `url`
+
+The URL of the target MongoDB server.
+
+*Type*: `string`
+
+*Default*: `""`
+
+```yml
+# Examples
+
+url: mongodb://localhost:27017
+```
+
+=== `database`
+
+The name of the MongoDB database to stream changes from.
+
+*Type*: `string`
+
+*Default*: `""`
+
+
+=== `username`
+
+The username to connect to the database.
+
+*Type*: `string`
+
+*Default*: `""`
+
+=== `password`
+
+The password to connect to the database.
+
+include::components:partial$secret_warning.adoc[]
+
+*Type*: `string`
+
+*Default*: `""`
+
+
+=== `collections`
+
+A list of collections to stream changes from. Specify each collection name as a separate item.
+
+*Type*: `array`
+
+*Default*: `[]`
+
+```yml
+# Examples
+
+collections:
+  - orders
+  - customers
+  - inventory
+```
+
+=== `checkpoint_cache`
+
+Specify a `cache` resource to store the oplog position for the most recent data update streamed to Redpanda Connect. After a restart, Redpanda Connect can continue processing changes from this position, avoiding the need to reprocess all collection updates.
+
+*Type*: `string`
+
+*Default*: `""`
+
+```yml
+# Examples
+
+input:
+  mongodb_cdc:
+    url: mongodb://localhost:27017
+    collections: [my_collection]
+    checkpoint_cache: "my_cdc_cache"
+cache_resources:
+  - label: "my_cdc_cache"
+    redis:
+      url: redis://:6379
+```
+
+=== `checkpoint_key`
+
+The key identifier used to store the oplog position in <<checkpoint_cache,`checkpoint_cache`>>. If you have multiple `mongodb_cdc` inputs sharing the same cache, you can provide an alternative key.
+
+
+*Type*: `string`
+
+*Default*: `mongodb_cdc_checkpoint`
+
+
+=== `checkpoint_interval`
+
+The interval between writing checkpoints to the cache.
+
+*Type*: `string`
+
+*Default*: `5s`
+
+=== `checkpoint_limit`
+
+The maximum number of messages that this input can process at a given time. Increasing this limit enables parallel processing, and batching at the output level. To preserve at-least-once guarantees, any given oplog position is not acknowledged until all messages under that offset are delivered.
path: modules/components/pages/inputs/mongodb_cdc.adoc
user: login: asimms41
body: Updated. I needed a verb so I used 'emitted'. Hope that is OK.
created_at: 2025-03-07T10:45:42Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,393 @@
+= mongodb_cdc
+// tag::single-source[]
+:type: input
+:page-beta: true
+:categories: ["Services"]
+
+component_type_dropdown::[]
+
+Streams data changes from a MongoDB replica set, using MongoDB's https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data updates.
+
+ifndef::env-cloud[]
+Introduced in version 4.48.1.
+endif::[]
+
+[tabs]
+======
+Common::
++
+--
+
+```yml
+# Configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: "" # Optional
+    password: "" # Optional
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    auto_replay_nacks: true
+```
+
+--
+Advanced::
++
+--
+
+```yml
+# All configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: ""
+    password: ""
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    snapshot_auto_bucket_sharding: false
+    document_mode: update_lookup
+    json_marshal_mode: canonical
+    app_name: benthos
+    auto_replay_nacks: true
+```
+--
+======
+
+== Prerequisites
+
+- MongoDB version 6 or later
+ifndef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment.
+endif::[]
+ifdef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment. For detailed networking information, including how to set up a VPC peering connection, see xref:networking:index.adoc[Redpanda Cloud Networking].
+endif::[]
+- A sharded MongoDB database or MongoDB cluster running as a replica set
+
+ifndef::env-cloud[]
+== Enable connectivity from cloud-based data sources
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the IP addresses of your Redpanda Connect instances to your firewall rules.
+
+endif::[]
+
+ifdef::env-cloud[]
+
+== Enable connectivity from cloud-based data sources (BYOC)
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the NAT Gateway IP address of your Redpanda cluster to the allowlist of your data source.
+
+endif::[]
+
+== Data capture method
+
+The `mongodb_cdc` input uses https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data changes, which does not propagate _all_ changes to Redpanda Connect. To capture all changes in a MongoDB cluster, including deletions, enable pre- and post-image saving for the cluster and <<database-collections, required collections>>. For more information, see <<document-mode, `document_mode` options>> and the https://www.mongodb.com/docs/manual/changeStreams/#change-streams-with-document-pre--and-post-images[MongoDB documentation].
+
+
+== Data replication
+
+Redpanda Connect allows you to specify which <<collections,database collections>> in your source database to receive changes from.
+
+You can also run the `mongodb_cdc` input in one of two modes, depending on whether you need a snapshot of existing data before streaming updates.
+
+- Snapshot mode: Redpanda Connect first captures a snapshot of all data in the selected collections and streams the contents before processing changes from the last recorded https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^] position.
+- Streaming mode: Redpanda Connect skips the snapshot and processes only the most recent data changes, starting from the latest oplog position.
+
+=== Snapshot mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `true`, Redpanda Connect connects to your MongoDB database and does the following to capture a snapshot of all data in the selected collections:
+
+//TODO should include parallel processing of snapshots and document_mode
+
+NOTE: If the pipeline restarts during this process, Redpanda Connect must start the snapshot capture from scratch to store the current oplog position in the <<checkpoint_cache,`checkpoint_cache`>>.
+
+//TODO
+
+Finally, the input uses the stored oplog position to catch up with changes that occurred during snapshot processing.
+
+=== Streaming mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `false`, Redpanda Connect connects to your MongoDB database and starts processing data changes from the latest oplog position. If the pipeline restarts, Redpanda Connect resumes processing updates from the last oplog position written to the <<checkpoint_cache,`checkpoint_cache`>>.
+
+== Data mappings
+
+The following table shows how selected MongoDB data types are mapped to data types supported in Redpanda Connect. All other data types are mapped to string values.
+
+|===
+| MongoDB data type | Bloblang value
+
+| Placeholder
+| Placeholder
+
+|===
+
+== Metadata
+
+This input adds the following metadata fields to each message:
+
+- `operation`: The type of data change that generated the message: `read`, `create`, `update`, `replace`, `delete`, `update`. A `read` operation occurs when the initial snapshot of the database is processed.
+- `collection`: The name of the collection from which the message originated.
+- `operation_time`: The time the data change was written to the https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^].
+
+== Fields
+
+=== `url`
+
+The URL of the target MongoDB server.
+
+*Type*: `string`
+
+*Default*: `""`
+
+```yml
+# Examples
+
+url: mongodb://localhost:27017
+```
+
+=== `database`
+
+The name of the MongoDB database to stream changes from.
+
+*Type*: `string`
+
+*Default*: `""`
+
+
+=== `username`
+
+The username to connect to the database.
+
+*Type*: `string`
+
+*Default*: `""`
+
+=== `password`
+
+The password to connect to the database.
+
+include::components:partial$secret_warning.adoc[]
+
+*Type*: `string`
+
+*Default*: `""`
+
+
+=== `collections`
+
+A list of collections to stream changes from. Specify each collection name as a separate item.
+
+*Type*: `array`
+
+*Default*: `[]`
+
+```yml
+# Examples
+
+collections:
+  - orders
+  - customers
+  - inventory
+```
+
+=== `checkpoint_cache`
+
+Specify a `cache` resource to store the oplog position for the most recent data update streamed to Redpanda Connect. After a restart, Redpanda Connect can continue processing changes from this position, avoiding the need to reprocess all collection updates.
+
+*Type*: `string`
+
+*Default*: `""`
+
+```yml
+# Examples
+
+input:
+  mongodb_cdc:
+    url: mongodb://localhost:27017
+    collections: [my_collection]
+    checkpoint_cache: "my_cdc_cache"
+cache_resources:
+  - label: "my_cdc_cache"
+    redis:
+      url: redis://:6379
+```
+
+=== `checkpoint_key`
+
+The key identifier used to store the oplog position in <<checkpoint_cache,`checkpoint_cache`>>. If you have multiple `mongodb_cdc` inputs sharing the same cache, you can provide an alternative key.
+
+
+*Type*: `string`
+
+*Default*: `mongodb_cdc_checkpoint`
+
+
+=== `checkpoint_interval`
+
+The interval between writing checkpoints to the cache.
+
+*Type*: `string`
+
+*Default*: `5s`
+
+=== `checkpoint_limit`
+
+The maximum number of messages that this input can process at a given time. Increasing this limit enables parallel processing, and batching at the output level. To preserve at-least-once guarantees, any given oplog position is not acknowledged until all messages under that offset are delivered.
+
+*Type*: `int`
+
+*Default*: `1000`
+
+
+=== `read_batch_size`
+
+The number of documents to fetch in each message batch from MongoDB.
+
+
+*Type*: `int`
+
+*Default*: `1000`
+
+
+=== `read_max_wait`
+
+The maximum duration MongoDB waits to accumulate the <<read-batch-size, `read_batch_size`>> documents on a change stream before returning the batch to Redpanda Connect.
+
+*Type*: `string`
+
+*Default*: `1s`
+
+=== `stream_snapshot`
+
+When set to `true`, this input streams a snapshot of all existing data in the source collections before streaming data changes.
+
+*Type*: `bool`
+
+*Default*: `false`
+
+```yml
+# Examples
+
+stream_snapshot: true
+```
+
+=== `snapshot_parallelism`
+
+Specifies the number of connections to use when reading the initial snapshot from one or more collections. Increase this number to enable parallel processing of the snapshot.
+
+This feature uses the `$splitVector` command to split snapshot data into chunks for more efficient processing.
+
+*Type*: `int`
+
+*Default*: `1`
+
+=== `snapshot_auto_bucket_sharding`
+
+Uses the https://www.mongodb.com/docs/manual/reference/operator/aggregation/bucketAuto/[`$bucketAuto`^] command instead of the default, `$splitVector`, to split the snapshot data into chunks for processing. This is required for environments, such as MongoDB Atlas, where privileged access is restricted. To enable parallel processing in these environments:
+
+- Set `snapshot_auto_bucket_sharding` to `true`.
+- Increase `snapshot_parallelism` to a value greater than `1`.
+
+*Type*: `bool`
+
+*Default*: `false`
+
+=== `document_mode`
+
+The mode in which MongoDB emits document changes to Redpanda Connect, specifically updates and deletes.
+
+[cols="1,2"]
+|===
+| Option | Mode description
+
+| `partial_update`
+a| Update operations contain only a description of the changes made to the document, using the following schema:
+
+[,bash,role="no-placeholders"]
+----
+{
+  "_id": "<document_id>",
+  "operations": [
+    # type == set: Indicates that a value was updated. For example:
+    # root.name."address.city" = "New York"
+    {"path": ["name", "address.city"], "type": "set", "value": "New York"},
+
+    # type == unset: Indicates that a value was deleted. For example:
+    # root.phone = deleted()
+    {"path": ["phone"], "type": "unset", "value": null},
+
+    # type == truncatedArray: Indicates that an array was truncated to a specified number of elements. For example:
+    # root.items = this.items.slice(2)
+    {"path": ["items"], "type": "truncatedArray", "value": 2}
+  ]
+}
+----
+
+| `pre_and_post_images`
+a| Emits full documents for update and delete operations by using pre- and post-image collections.  
+For setup and configuration details, see the https://www.mongodb.com/docs/manual/changeStreams/#change-streams-with-document-pre--and-post-images[MongoDB documentation^].
+
+| `update_lookup`
+a| Emits full documents for insert, replace, and update operations, but only the `_id` field for delete operations.  
+This corresponds to the `updateLookup` option.  
+For more information, see the https://www.mongodb.com/docs/manual/changeStreams/#std-label-change-streams-updateLookup[MongoDB documentation^].
+
+|===
+
+=== `json_marshal_mode`
+
+Controls how MongoDB formats messages sent to Redpanda Connect.
+
+*Type*: `string`
+
+*Default*: `canonical`
+
+|===
+| Option | Description
+
+| `canonical`
+| A string format that preserves type information at the expense of readability and interoperability. Conversion from canonical to Binary JSON (BSON) generally retains type information, except in specific cases.
path: modules/components/pages/inputs/mongodb_cdc.adoc
user: login: asimms41
body: Updated.
created_at: 2025-03-07T10:46:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,393 @@
+= mongodb_cdc
+// tag::single-source[]
+:type: input
+:page-beta: true
+:categories: ["Services"]
+
+component_type_dropdown::[]
+
+Streams data changes from a MongoDB replica set, using MongoDB's https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data updates.
+
+ifndef::env-cloud[]
+Introduced in version 4.48.1.
+endif::[]
+
+[tabs]
+======
+Common::
++
+--
+
+```yml
+# Configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: "" # Optional
+    password: "" # Optional
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    auto_replay_nacks: true
+```
+
+--
+Advanced::
++
+--
+
+```yml
+# All configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: ""
+    password: ""
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    snapshot_auto_bucket_sharding: false
+    document_mode: update_lookup
+    json_marshal_mode: canonical
+    app_name: benthos
+    auto_replay_nacks: true
+```
+--
+======
+
+== Prerequisites
+
+- MongoDB version 6 or later
+ifndef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment.
+endif::[]
+ifdef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment. For detailed networking information, including how to set up a VPC peering connection, see xref:networking:index.adoc[Redpanda Cloud Networking].
+endif::[]
+- A sharded MongoDB database or MongoDB cluster running as a replica set
path: modules/components/pages/inputs/mongodb_cdc.adoc
user: login: asimms41
body: Updated.
created_at: 2025-03-07T10:46:18Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,393 @@
+= mongodb_cdc
+// tag::single-source[]
+:type: input
+:page-beta: true
+:categories: ["Services"]
+
+component_type_dropdown::[]
+
+Streams data changes from a MongoDB replica set, using MongoDB's https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data updates.
+
+ifndef::env-cloud[]
+Introduced in version 4.48.1.
+endif::[]
+
+[tabs]
+======
+Common::
++
+--
+
+```yml
+# Configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: "" # Optional
+    password: "" # Optional
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    auto_replay_nacks: true
+```
+
+--
+Advanced::
++
+--
+
+```yml
+# All configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: ""
+    password: ""
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    snapshot_auto_bucket_sharding: false
+    document_mode: update_lookup
+    json_marshal_mode: canonical
+    app_name: benthos
+    auto_replay_nacks: true
+```
+--
+======
+
+== Prerequisites
+
+- MongoDB version 6 or later
+ifndef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment.
+endif::[]
+ifdef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment. For detailed networking information, including how to set up a VPC peering connection, see xref:networking:index.adoc[Redpanda Cloud Networking].
+endif::[]
+- A sharded MongoDB database or MongoDB cluster running as a replica set
+
+ifndef::env-cloud[]
+== Enable connectivity from cloud-based data sources
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the IP addresses of your Redpanda Connect instances to your firewall rules.
+
+endif::[]
+
+ifdef::env-cloud[]
+
+== Enable connectivity from cloud-based data sources (BYOC)
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the NAT Gateway IP address of your Redpanda cluster to the allowlist of your data source.
+
+endif::[]
+
+== Data capture method
+
+The `mongodb_cdc` input uses https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data changes, which does not propagate _all_ changes to Redpanda Connect. To capture all changes in a MongoDB cluster, including deletions, enable pre- and post-image saving for the cluster and <<database-collections, required collections>>. For more information, see <<document-mode, `document_mode` options>> and the https://www.mongodb.com/docs/manual/changeStreams/#change-streams-with-document-pre--and-post-images[MongoDB documentation].
+
+
+== Data replication
+
+Redpanda Connect allows you to specify which <<collections,database collections>> in your source database to receive changes from.
+
+You can also run the `mongodb_cdc` input in one of two modes, depending on whether you need a snapshot of existing data before streaming updates.
+
+- Snapshot mode: Redpanda Connect first captures a snapshot of all data in the selected collections and streams the contents before processing changes from the last recorded https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^] position.
+- Streaming mode: Redpanda Connect skips the snapshot and processes only the most recent data changes, starting from the latest oplog position.
+
+=== Snapshot mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `true`, Redpanda Connect connects to your MongoDB database and does the following to capture a snapshot of all data in the selected collections:
+
+//TODO should include parallel processing of snapshots and document_mode
+
+NOTE: If the pipeline restarts during this process, Redpanda Connect must start the snapshot capture from scratch to store the current oplog position in the <<checkpoint_cache,`checkpoint_cache`>>.
+
+//TODO
path: modules/components/pages/inputs/mongodb_cdc.adoc
user: login: asimms41
body: Updated.
created_at: 2025-03-07T10:48:29Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,393 @@
+= mongodb_cdc
+// tag::single-source[]
+:type: input
+:page-beta: true
+:categories: ["Services"]
+
+component_type_dropdown::[]
+
+Streams data changes from a MongoDB replica set, using MongoDB's https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data updates.
+
+ifndef::env-cloud[]
+Introduced in version 4.48.1.
+endif::[]
+
+[tabs]
+======
+Common::
++
+--
+
+```yml
+# Configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: "" # Optional
+    password: "" # Optional
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    auto_replay_nacks: true
+```
+
+--
+Advanced::
++
+--
+
+```yml
+# All configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: ""
+    password: ""
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    snapshot_auto_bucket_sharding: false
+    document_mode: update_lookup
+    json_marshal_mode: canonical
+    app_name: benthos
+    auto_replay_nacks: true
+```
+--
+======
+
+== Prerequisites
+
+- MongoDB version 6 or later
+ifndef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment.
+endif::[]
+ifdef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment. For detailed networking information, including how to set up a VPC peering connection, see xref:networking:index.adoc[Redpanda Cloud Networking].
+endif::[]
+- A sharded MongoDB database or MongoDB cluster running as a replica set
+
+ifndef::env-cloud[]
+== Enable connectivity from cloud-based data sources
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the IP addresses of your Redpanda Connect instances to your firewall rules.
+
+endif::[]
+
+ifdef::env-cloud[]
+
+== Enable connectivity from cloud-based data sources (BYOC)
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the NAT Gateway IP address of your Redpanda cluster to the allowlist of your data source.
+
+endif::[]
+
+== Data capture method
+
+The `mongodb_cdc` input uses https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data changes, which does not propagate _all_ changes to Redpanda Connect. To capture all changes in a MongoDB cluster, including deletions, enable pre- and post-image saving for the cluster and <<database-collections, required collections>>. For more information, see <<document-mode, `document_mode` options>> and the https://www.mongodb.com/docs/manual/changeStreams/#change-streams-with-document-pre--and-post-images[MongoDB documentation].
+
+
+== Data replication
+
+Redpanda Connect allows you to specify which <<collections,database collections>> in your source database to receive changes from.
+
+You can also run the `mongodb_cdc` input in one of two modes, depending on whether you need a snapshot of existing data before streaming updates.
+
+- Snapshot mode: Redpanda Connect first captures a snapshot of all data in the selected collections and streams the contents before processing changes from the last recorded https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^] position.
+- Streaming mode: Redpanda Connect skips the snapshot and processes only the most recent data changes, starting from the latest oplog position.
+
+=== Snapshot mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `true`, Redpanda Connect connects to your MongoDB database and does the following to capture a snapshot of all data in the selected collections:
+
+//TODO should include parallel processing of snapshots and document_mode
+
+NOTE: If the pipeline restarts during this process, Redpanda Connect must start the snapshot capture from scratch to store the current oplog position in the <<checkpoint_cache,`checkpoint_cache`>>.
+
+//TODO
+
+Finally, the input uses the stored oplog position to catch up with changes that occurred during snapshot processing.
+
+=== Streaming mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `false`, Redpanda Connect connects to your MongoDB database and starts processing data changes from the latest oplog position. If the pipeline restarts, Redpanda Connect resumes processing updates from the last oplog position written to the <<checkpoint_cache,`checkpoint_cache`>>.
+
+== Data mappings
+
+The following table shows how selected MongoDB data types are mapped to data types supported in Redpanda Connect. All other data types are mapped to string values.
+
+|===
+| MongoDB data type | Bloblang value
+
+| Placeholder
+| Placeholder
+
+|===
+
+== Metadata
+
+This input adds the following metadata fields to each message:
+
+- `operation`: The type of data change that generated the message: `read`, `create`, `update`, `replace`, `delete`, `update`. A `read` operation occurs when the initial snapshot of the database is processed.
+- `collection`: The name of the collection from which the message originated.
+- `operation_time`: The time the data change was written to the https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^].
+
+== Fields
+
+=== `url`
+
+The URL of the target MongoDB server.
+
+*Type*: `string`
+
+*Default*: `""`
+
+```yml
+# Examples
+
+url: mongodb://localhost:27017
+```
+
+=== `database`
+
+The name of the MongoDB database to stream changes from.
+
+*Type*: `string`
+
+*Default*: `""`
+
+
+=== `username`
+
+The username to connect to the database.
+
+*Type*: `string`
+
+*Default*: `""`
+
+=== `password`
+
+The password to connect to the database.
+
+include::components:partial$secret_warning.adoc[]
+
+*Type*: `string`
+
+*Default*: `""`
+
+
+=== `collections`
+
+A list of collections to stream changes from. Specify each collection name as a separate item.
+
+*Type*: `array`
+
+*Default*: `[]`
+
+```yml
+# Examples
+
+collections:
+  - orders
+  - customers
+  - inventory
+```
+
+=== `checkpoint_cache`
+
+Specify a `cache` resource to store the oplog position for the most recent data update streamed to Redpanda Connect. After a restart, Redpanda Connect can continue processing changes from this position, avoiding the need to reprocess all collection updates.
+
+*Type*: `string`
+
+*Default*: `""`
+
+```yml
+# Examples
+
+input:
+  mongodb_cdc:
+    url: mongodb://localhost:27017
+    collections: [my_collection]
+    checkpoint_cache: "my_cdc_cache"
+cache_resources:
+  - label: "my_cdc_cache"
+    redis:
+      url: redis://:6379
+```
+
+=== `checkpoint_key`
+
+The key identifier used to store the oplog position in <<checkpoint_cache,`checkpoint_cache`>>. If you have multiple `mongodb_cdc` inputs sharing the same cache, you can provide an alternative key.
+
+
+*Type*: `string`
+
+*Default*: `mongodb_cdc_checkpoint`
+
+
+=== `checkpoint_interval`
+
+The interval between writing checkpoints to the cache.
+
+*Type*: `string`
+
+*Default*: `5s`
+
+=== `checkpoint_limit`
+
+The maximum number of messages that this input can process at a given time. Increasing this limit enables parallel processing, and batching at the output level. To preserve at-least-once guarantees, any given oplog position is not acknowledged until all messages under that offset are delivered.
+
+*Type*: `int`
+
+*Default*: `1000`
+
+
+=== `read_batch_size`
+
+The number of documents to fetch in each message batch from MongoDB.
+
+
+*Type*: `int`
+
+*Default*: `1000`
+
+
+=== `read_max_wait`
+
+The maximum duration MongoDB waits to accumulate the <<read-batch-size, `read_batch_size`>> documents on a change stream before returning the batch to Redpanda Connect.
+
+*Type*: `string`
+
+*Default*: `1s`
+
+=== `stream_snapshot`
+
+When set to `true`, this input streams a snapshot of all existing data in the source collections before streaming data changes.
+
+*Type*: `bool`
+
+*Default*: `false`
+
+```yml
+# Examples
+
+stream_snapshot: true
+```
+
+=== `snapshot_parallelism`
+
+Specifies the number of connections to use when reading the initial snapshot from one or more collections. Increase this number to enable parallel processing of the snapshot.
+
+This feature uses the `$splitVector` command to split snapshot data into chunks for more efficient processing.
+
+*Type*: `int`
+
+*Default*: `1`
+
+=== `snapshot_auto_bucket_sharding`
+
+Uses the https://www.mongodb.com/docs/manual/reference/operator/aggregation/bucketAuto/[`$bucketAuto`^] command instead of the default, `$splitVector`, to split the snapshot data into chunks for processing. This is required for environments, such as MongoDB Atlas, where privileged access is restricted. To enable parallel processing in these environments:
+
+- Set `snapshot_auto_bucket_sharding` to `true`.
+- Increase `snapshot_parallelism` to a value greater than `1`.
+
+*Type*: `bool`
+
+*Default*: `false`
+
+=== `document_mode`
+
+The mode in which MongoDB emits document changes to Redpanda Connect, specifically updates and deletes.
+
+[cols="1,2"]
+|===
+| Option | Mode description
+
+| `partial_update`
+a| Update operations contain only a description of the changes made to the document, using the following schema:
+
+[,bash,role="no-placeholders"]
+----
+{
+  "_id": "<document_id>",
+  "operations": [
+    # type == set: Indicates that a value was updated. For example:
+    # root.name."address.city" = "New York"
+    {"path": ["name", "address.city"], "type": "set", "value": "New York"},
+
+    # type == unset: Indicates that a value was deleted. For example:
+    # root.phone = deleted()
+    {"path": ["phone"], "type": "unset", "value": null},
+
+    # type == truncatedArray: Indicates that an array was truncated to a specified number of elements. For example:
+    # root.items = this.items.slice(2)
+    {"path": ["items"], "type": "truncatedArray", "value": 2}
+  ]
+}
+----
+
+| `pre_and_post_images`
+a| Emits full documents for update and delete operations by using pre- and post-image collections.  
+For setup and configuration details, see the https://www.mongodb.com/docs/manual/changeStreams/#change-streams-with-document-pre--and-post-images[MongoDB documentation^].
+
+| `update_lookup`
+a| Emits full documents for insert, replace, and update operations, but only the `_id` field for delete operations.  
+This corresponds to the `updateLookup` option.  
+For more information, see the https://www.mongodb.com/docs/manual/changeStreams/#std-label-change-streams-updateLookup[MongoDB documentation^].
+
+|===
+
+=== `json_marshal_mode`
+
+Controls the format of the message output to Redpanda Connect.
+
+*Type*: `string`
+
+*Default*: `canonical`
+
+|===
+| Option | Description
+
+| `canonical`
+| A string format that preserves the exact data types of values stored in MongoDB at the expense of readability and interoperability. Conversion from canonical to Binary JSON (BSON) generally retains type information, except in specific cases.
path: modules/components/pages/inputs/mongodb_cdc.adoc
user: login: asimms41
body: Updated. Let me know what you think.
created_at: 2025-03-07T10:49:06Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,392 @@
+= mongodb_cdc
+// tag::single-source[]
+:type: input
+:page-beta: true
+:categories: ["Services"]
+
+component_type_dropdown::[]
+
+Streams data changes from a MongoDB replica set, using MongoDB's https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data updates.
+
+ifndef::env-cloud[]
+Introduced in version 4.48.1.
+endif::[]
+
+[tabs]
+======
+Common::
++
+--
+
+```yml
+# Configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: "" # Optional
+    password: "" # Optional
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    auto_replay_nacks: true
+```
+
+--
+Advanced::
++
+--
+
+```yml
+# All configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: ""
+    password: ""
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    snapshot_auto_bucket_sharding: false
+    document_mode: update_lookup
+    json_marshal_mode: canonical
+    app_name: benthos
+    auto_replay_nacks: true
+```
+--
+======
+
+== Prerequisites
+
+- MongoDB version 6 or later
+ifndef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment.
+endif::[]
+ifdef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment. For detailed networking information, including how to set up a VPC peering connection, see xref:networking:index.adoc[Redpanda Cloud Networking].
+endif::[]
+- A MongoDB database running as a https://www.mongodb.com/docs/manual/replication/#replication-in-mongodb[replica set^] or in a https://www.mongodb.com/docs/manual/sharding/[sharded cluster^] using replica set https://www.mongodb.com/docs/manual/reference/replica-configuration/#rsconf.protocolVersion[protocol version 1].
+- A MongoDB database using the https://www.mongodb.com/docs/manual/core/wiredtiger/#storage-wiredtiger[WiredTiger] storage engine.
+
+
+ifndef::env-cloud[]
+== Enable connectivity from cloud-based data sources
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the IP addresses of your Redpanda Connect instances to your firewall rules.
+
+endif::[]
+
+ifdef::env-cloud[]
+
+== Enable connectivity from cloud-based data sources (BYOC)
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the NAT Gateway IP address of your Redpanda cluster to the allowlist of your data source.
+
+endif::[]
+
+== Data capture method
+
+The `mongodb_cdc` input uses https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data changes, which does not propagate _all_ changes to Redpanda Connect. To capture all changes in a MongoDB cluster, including deletions, enable pre- and post-image saving for the cluster and <<collections,required collections>>. For more information, see <<document_mode,`document_mode` options>> and the https://www.mongodb.com/docs/manual/changeStreams/#change-streams-with-document-pre--and-post-images[MongoDB documentation].
+
+
+== Data replication
+
+Redpanda Connect allows you to specify which <<collections,database collections>> in your source database to receive changes from.
+
+You can also run the `mongodb_cdc` input in one of two modes, depending on whether you need a snapshot of existing data before streaming updates.
+
+- Snapshot mode: Redpanda Connect first captures a snapshot of all data in the selected collections and streams the contents before processing changes from the last recorded https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^] position.
+- Streaming mode: Redpanda Connect skips the snapshot and processes only the most recent data changes, starting from the latest oplog position.
+
+=== Snapshot mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `true`, Redpanda Connect connects to your MongoDB database and does the following to capture a snapshot of all data in the selected collections:
+
+. Records the latest oplog position.
+. Determines the strategy for splitting the snapshot data down into shards or chunks for more efficient processing:
+.. If <<snapshot_auto_bucket_sharding,`snapshot_auto_bucket_sharding`>> is set to `false`, the internal `$splitVector` command is used to compute shards.  
+.. If <<snapshot_auto_bucket_sharding,`snapshot_auto_bucket_sharding`>> is set to `true`, the https://www.mongodb.com/docs/manual/reference/operator/aggregation/bucketAuto/[`$bucketAuto`^] command is used instead. This setting is for environments, such as MongoDB Atlas, where privileged access is restricted.
+. This input then uses the number of connections specified in <<snapshot_parallelism,`snapshot-parallelism`>> to read the selected collections.
++
+NOTE: If the pipeline restarts during this process, Redpanda Connect must start the snapshot capture from scratch to store the current oplog position in the <<checkpoint_cache,`checkpoint_cache`>>.
+. Finally, the input uses the stored oplog position to catch up with changes that occurred during snapshot processing.
+
+=== Streaming mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `false`, Redpanda Connect connects to your MongoDB database and starts processing data changes from the latest oplog position. If the pipeline restarts, Redpanda Connect resumes processing updates from the last oplog position written to the <<checkpoint_cache,`checkpoint_cache`>>.
+
+== Metadata
+
+This input adds the following metadata fields to each message:
+
+- `operation`: The type of data change that generated the message: `read`, `create`, `update`, `replace`, `delete`, `update`. A `read` operation occurs when the initial snapshot of the database is processed.
+- `collection`: The name of the collection from which the message originated.
+- `operation_time`: The time the data change was written to the https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^] in the form of a Binary JSON (BSON) timestamp: `{"t": <unix seconds>, "i": <sequence number>}`.
+
+== Fields
+
+=== `url`
+
+The URL of the target MongoDB server.
+
+*Type*: `string`
+
+*Default*: `""`
+
+```yml
+# Examples
+
+url: mongodb://localhost:27017
+```
+
+=== `database`
+
+The name of the MongoDB database to stream changes from.
+
+*Type*: `string`
+
+*Default*: `""`
+
+
+=== `username`
+
+The username to connect to the database.
+
+*Type*: `string`
+
+*Default*: `""`
+
+=== `password`
+
+The password to connect to the database.
+
+include::components:partial$secret_warning.adoc[]
+
+*Type*: `string`
+
+*Default*: `""`
+
+
+=== `collections`
+
+A list of collections to stream changes from. Specify each collection name as a separate item.
+
+*Type*: `array`
+
+*Default*: `[]`
+
+```yml
+# Examples
+
+collections:
+  - orders
+  - customers
+  - inventory
+```
+
+=== `checkpoint_cache`
+
+Specify a `cache` resource to store the oplog position for the most recent data update streamed to Redpanda Connect. After a restart, Redpanda Connect can continue processing changes from this position, avoiding the need to reprocess all collection updates.
path: modules/components/pages/inputs/mongodb_cdc.adoc
user: login: rockwotj
body: Should we link to the cache documentation here?
created_at: 2025-03-07T15:54:25Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,392 @@
+= mongodb_cdc
+// tag::single-source[]
+:type: input
+:page-beta: true
+:categories: ["Services"]
+
+component_type_dropdown::[]
+
+Streams data changes from a MongoDB replica set, using MongoDB's https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data updates.
+
+ifndef::env-cloud[]
+Introduced in version 4.48.1.
+endif::[]
+
+[tabs]
+======
+Common::
++
+--
+
+```yml
+# Configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: "" # Optional
+    password: "" # Optional
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    auto_replay_nacks: true
+```
+
+--
+Advanced::
++
+--
+
+```yml
+# All configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: ""
+    password: ""
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    snapshot_auto_bucket_sharding: false
+    document_mode: update_lookup
+    json_marshal_mode: canonical
+    app_name: benthos
+    auto_replay_nacks: true
+```
+--
+======
+
+== Prerequisites
+
+- MongoDB version 6 or later
+ifndef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment.
+endif::[]
+ifdef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment. For detailed networking information, including how to set up a VPC peering connection, see xref:networking:index.adoc[Redpanda Cloud Networking].
+endif::[]
+- A MongoDB database running as a https://www.mongodb.com/docs/manual/replication/#replication-in-mongodb[replica set^] or in a https://www.mongodb.com/docs/manual/sharding/[sharded cluster^] using replica set https://www.mongodb.com/docs/manual/reference/replica-configuration/#rsconf.protocolVersion[protocol version 1].
+- A MongoDB database using the https://www.mongodb.com/docs/manual/core/wiredtiger/#storage-wiredtiger[WiredTiger] storage engine.
+
+
+ifndef::env-cloud[]
+== Enable connectivity from cloud-based data sources
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the IP addresses of your Redpanda Connect instances to your firewall rules.
+
+endif::[]
+
+ifdef::env-cloud[]
+
+== Enable connectivity from cloud-based data sources (BYOC)
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the NAT Gateway IP address of your Redpanda cluster to the allowlist of your data source.
+
+endif::[]
+
+== Data capture method
+
+The `mongodb_cdc` input uses https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data changes, which does not propagate _all_ changes to Redpanda Connect. To capture all changes in a MongoDB cluster, including deletions, enable pre- and post-image saving for the cluster and <<collections,required collections>>. For more information, see <<document_mode,`document_mode` options>> and the https://www.mongodb.com/docs/manual/changeStreams/#change-streams-with-document-pre--and-post-images[MongoDB documentation].
+
+
+== Data replication
+
+Redpanda Connect allows you to specify which <<collections,database collections>> in your source database to receive changes from.
+
+You can also run the `mongodb_cdc` input in one of two modes, depending on whether you need a snapshot of existing data before streaming updates.
+
+- Snapshot mode: Redpanda Connect first captures a snapshot of all data in the selected collections and streams the contents before processing changes from the last recorded https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^] position.
+- Streaming mode: Redpanda Connect skips the snapshot and processes only the most recent data changes, starting from the latest oplog position.
+
+=== Snapshot mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `true`, Redpanda Connect connects to your MongoDB database and does the following to capture a snapshot of all data in the selected collections:
+
+. Records the latest oplog position.
+. Determines the strategy for splitting the snapshot data down into shards or chunks for more efficient processing:
+.. If <<snapshot_auto_bucket_sharding,`snapshot_auto_bucket_sharding`>> is set to `false`, the internal `$splitVector` command is used to compute shards.  
+.. If <<snapshot_auto_bucket_sharding,`snapshot_auto_bucket_sharding`>> is set to `true`, the https://www.mongodb.com/docs/manual/reference/operator/aggregation/bucketAuto/[`$bucketAuto`^] command is used instead. This setting is for environments, such as MongoDB Atlas, where privileged access is restricted.
+. This input then uses the number of connections specified in <<snapshot_parallelism,`snapshot-parallelism`>> to read the selected collections.
++
+NOTE: If the pipeline restarts during this process, Redpanda Connect must start the snapshot capture from scratch to store the current oplog position in the <<checkpoint_cache,`checkpoint_cache`>>.
+. Finally, the input uses the stored oplog position to catch up with changes that occurred during snapshot processing.
+
+=== Streaming mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `false`, Redpanda Connect connects to your MongoDB database and starts processing data changes from the latest oplog position. If the pipeline restarts, Redpanda Connect resumes processing updates from the last oplog position written to the <<checkpoint_cache,`checkpoint_cache`>>.
+
+== Metadata
+
+This input adds the following metadata fields to each message:
+
+- `operation`: The type of data change that generated the message: `read`, `create`, `update`, `replace`, `delete`, `update`. A `read` operation occurs when the initial snapshot of the database is processed.
+- `collection`: The name of the collection from which the message originated.
+- `operation_time`: The time the data change was written to the https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^] in the form of a Binary JSON (BSON) timestamp: `{"t": <unix seconds>, "i": <sequence number>}`.
+
+== Fields
+
+=== `url`
+
+The URL of the target MongoDB server.
+
+*Type*: `string`
+
+*Default*: `""`
+
+```yml
+# Examples
+
+url: mongodb://localhost:27017
+```
+
+=== `database`
+
+The name of the MongoDB database to stream changes from.
+
+*Type*: `string`
+
+*Default*: `""`
+
+
+=== `username`
+
+The username to connect to the database.
+
+*Type*: `string`
+
+*Default*: `""`
+
+=== `password`
+
+The password to connect to the database.
+
+include::components:partial$secret_warning.adoc[]
+
+*Type*: `string`
+
+*Default*: `""`
+
+
+=== `collections`
+
+A list of collections to stream changes from. Specify each collection name as a separate item.
+
+*Type*: `array`
+
+*Default*: `[]`
+
+```yml
+# Examples
+
+collections:
+  - orders
+  - customers
+  - inventory
+```
+
+=== `checkpoint_cache`
+
+Specify a `cache` resource to store the oplog position for the most recent data update streamed to Redpanda Connect. After a restart, Redpanda Connect can continue processing changes from this position, avoiding the need to reprocess all collection updates.
+
+*Type*: `string`
+
+*Default*: `""`
+
+```yml
+# Examples
+
+input:
+  mongodb_cdc:
+    url: mongodb://localhost:27017
+    collections: [my_collection]
+    checkpoint_cache: "my_cdc_cache"
+cache_resources:
+  - label: "my_cdc_cache"
+    redis:
+      url: redis://:6379
+```
+
+=== `checkpoint_key`
+
+The key identifier used to store the oplog position in <<checkpoint_cache,`checkpoint_cache`>>. If you have multiple `mongodb_cdc` inputs sharing the same cache, you can provide an alternative key.
+
+
+*Type*: `string`
+
+*Default*: `mongodb_cdc_checkpoint`
+
+
+=== `checkpoint_interval`
+
+The interval between writing checkpoints to the cache.
+
+*Type*: `string`
+
+*Default*: `5s`
+
+=== `checkpoint_limit`
+
+The maximum number of in-flight messages emitted from this input. Increasing this limit enables parallel processing, and batching at the output level. To preserve at-least-once guarantees, any given oplog position is not acknowledged until all messages under that offset are delivered.
+
+*Type*: `int`
+
+*Default*: `1000`
+
+
+=== `read_batch_size`
+
+The number of documents to fetch in each message batch from MongoDB.
+
+
+*Type*: `int`
+
+*Default*: `1000`
+
+
+=== `read_max_wait`
+
+The maximum duration MongoDB waits to accumulate the <<read_batch_size,`read_batch_size`>> documents on a change stream before returning the batch to Redpanda Connect.
+
+*Type*: `string`
+
+*Default*: `1s`
+
+=== `stream_snapshot`
+
+When set to `true`, this input streams a snapshot of all existing data in the source collections before streaming data changes.
+
+*Type*: `bool`
+
+*Default*: `false`
+
+```yml
+# Examples
+
+stream_snapshot: true
+```
+
+=== `snapshot_parallelism`
+
+Specifies the number of connections to use when reading the initial snapshot from one or more collections. Increase this number to enable parallel processing of the snapshot. 
+
+This feature uses the `$splitVector` command to split snapshot data into chunks for more efficient processing.
+
+This field is only applicable when `stream_snapshot` is set to `true`. 
+
+*Type*: `int`
+
+*Default*: `1`
+
+=== `snapshot_auto_bucket_sharding`
+
+Uses the https://www.mongodb.com/docs/manual/reference/operator/aggregation/bucketAuto/[`$bucketAuto`^] command instead of the default, `$splitVector`, to split the snapshot data into chunks for processing. This is required for environments, such as MongoDB Atlas, where privileged access is restricted. To enable parallel processing in these environments:
path: modules/components/pages/inputs/mongodb_cdc.adoc
user: login: rockwotj
body: ```suggestion
Uses the https://www.mongodb.com/docs/manual/reference/operator/aggregation/bucketAuto/[`$bucketAuto`^] command instead of the default, `$splitVector`, to split the snapshot data into chunks for processing. This is required for environments, such as MongoDB Atlas, where the `$splitVector` command is not available. To enable parallel processing in these environments:
```
created_at: 2025-03-07T15:57:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,392 @@
+= mongodb_cdc
+// tag::single-source[]
+:type: input
+:page-beta: true
+:categories: ["Services"]
+
+component_type_dropdown::[]
+
+Streams data changes from a MongoDB replica set, using MongoDB's https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data updates.
+
+ifndef::env-cloud[]
+Introduced in version 4.48.1.
+endif::[]
+
+[tabs]
+======
+Common::
++
+--
+
+```yml
+# Configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: "" # Optional
+    password: "" # Optional
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    auto_replay_nacks: true
+```
+
+--
+Advanced::
++
+--
+
+```yml
+# All configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: ""
+    password: ""
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    snapshot_auto_bucket_sharding: false
+    document_mode: update_lookup
+    json_marshal_mode: canonical
+    app_name: benthos
+    auto_replay_nacks: true
+```
+--
+======
+
+== Prerequisites
+
+- MongoDB version 6 or later
+ifndef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment.
+endif::[]
+ifdef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment. For detailed networking information, including how to set up a VPC peering connection, see xref:networking:index.adoc[Redpanda Cloud Networking].
+endif::[]
+- A MongoDB database running as a https://www.mongodb.com/docs/manual/replication/#replication-in-mongodb[replica set^] or in a https://www.mongodb.com/docs/manual/sharding/[sharded cluster^] using replica set https://www.mongodb.com/docs/manual/reference/replica-configuration/#rsconf.protocolVersion[protocol version 1].
+- A MongoDB database using the https://www.mongodb.com/docs/manual/core/wiredtiger/#storage-wiredtiger[WiredTiger] storage engine.
+
+
+ifndef::env-cloud[]
+== Enable connectivity from cloud-based data sources
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the IP addresses of your Redpanda Connect instances to your firewall rules.
+
+endif::[]
+
+ifdef::env-cloud[]
+
+== Enable connectivity from cloud-based data sources (BYOC)
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the NAT Gateway IP address of your Redpanda cluster to the allowlist of your data source.
+
+endif::[]
+
+== Data capture method
+
+The `mongodb_cdc` input uses https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data changes, which does not propagate _all_ changes to Redpanda Connect. To capture all changes in a MongoDB cluster, including deletions, enable pre- and post-image saving for the cluster and <<collections,required collections>>. For more information, see <<document_mode,`document_mode` options>> and the https://www.mongodb.com/docs/manual/changeStreams/#change-streams-with-document-pre--and-post-images[MongoDB documentation].
+
+
+== Data replication
+
+Redpanda Connect allows you to specify which <<collections,database collections>> in your source database to receive changes from.
+
+You can also run the `mongodb_cdc` input in one of two modes, depending on whether you need a snapshot of existing data before streaming updates.
+
+- Snapshot mode: Redpanda Connect first captures a snapshot of all data in the selected collections and streams the contents before processing changes from the last recorded https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^] position.
+- Streaming mode: Redpanda Connect skips the snapshot and processes only the most recent data changes, starting from the latest oplog position.
+
+=== Snapshot mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `true`, Redpanda Connect connects to your MongoDB database and does the following to capture a snapshot of all data in the selected collections:
+
+. Records the latest oplog position.
+. Determines the strategy for splitting the snapshot data down into shards or chunks for more efficient processing:
+.. If <<snapshot_auto_bucket_sharding,`snapshot_auto_bucket_sharding`>> is set to `false`, the internal `$splitVector` command is used to compute shards.  
+.. If <<snapshot_auto_bucket_sharding,`snapshot_auto_bucket_sharding`>> is set to `true`, the https://www.mongodb.com/docs/manual/reference/operator/aggregation/bucketAuto/[`$bucketAuto`^] command is used instead. This setting is for environments, such as MongoDB Atlas, where privileged access is restricted.
+. This input then uses the number of connections specified in <<snapshot_parallelism,`snapshot-parallelism`>> to read the selected collections.
++
+NOTE: If the pipeline restarts during this process, Redpanda Connect must start the snapshot capture from scratch to store the current oplog position in the <<checkpoint_cache,`checkpoint_cache`>>.
+. Finally, the input uses the stored oplog position to catch up with changes that occurred during snapshot processing.
+
+=== Streaming mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `false`, Redpanda Connect connects to your MongoDB database and starts processing data changes from the latest oplog position. If the pipeline restarts, Redpanda Connect resumes processing updates from the last oplog position written to the <<checkpoint_cache,`checkpoint_cache`>>.
+
+== Metadata
+
+This input adds the following metadata fields to each message:
+
+- `operation`: The type of data change that generated the message: `read`, `create`, `update`, `replace`, `delete`, `update`. A `read` operation occurs when the initial snapshot of the database is processed.
+- `collection`: The name of the collection from which the message originated.
+- `operation_time`: The time the data change was written to the https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^] in the form of a Binary JSON (BSON) timestamp: `{"t": <unix seconds>, "i": <sequence number>}`.
+
+== Fields
+
+=== `url`
+
+The URL of the target MongoDB server.
+
+*Type*: `string`
+
+*Default*: `""`
+
+```yml
+# Examples
+
+url: mongodb://localhost:27017
+```
+
+=== `database`
+
+The name of the MongoDB database to stream changes from.
+
+*Type*: `string`
+
+*Default*: `""`
+
+
+=== `username`
+
+The username to connect to the database.
+
+*Type*: `string`
+
+*Default*: `""`
+
+=== `password`
+
+The password to connect to the database.
+
+include::components:partial$secret_warning.adoc[]
+
+*Type*: `string`
+
+*Default*: `""`
+
+
+=== `collections`
+
+A list of collections to stream changes from. Specify each collection name as a separate item.
+
+*Type*: `array`
+
+*Default*: `[]`
+
+```yml
+# Examples
+
+collections:
+  - orders
+  - customers
+  - inventory
+```
+
+=== `checkpoint_cache`
+
+Specify a `cache` resource to store the oplog position for the most recent data update streamed to Redpanda Connect. After a restart, Redpanda Connect can continue processing changes from this position, avoiding the need to reprocess all collection updates.
+
+*Type*: `string`
+
+*Default*: `""`
+
+```yml
+# Examples
+
+input:
+  mongodb_cdc:
+    url: mongodb://localhost:27017
+    collections: [my_collection]
+    checkpoint_cache: "my_cdc_cache"
+cache_resources:
+  - label: "my_cdc_cache"
+    redis:
+      url: redis://:6379
+```
+
+=== `checkpoint_key`
+
+The key identifier used to store the oplog position in <<checkpoint_cache,`checkpoint_cache`>>. If you have multiple `mongodb_cdc` inputs sharing the same cache, you can provide an alternative key.
+
+
+*Type*: `string`
+
+*Default*: `mongodb_cdc_checkpoint`
+
+
+=== `checkpoint_interval`
+
+The interval between writing checkpoints to the cache.
+
+*Type*: `string`
+
+*Default*: `5s`
+
+=== `checkpoint_limit`
+
+The maximum number of in-flight messages emitted from this input. Increasing this limit enables parallel processing, and batching at the output level. To preserve at-least-once guarantees, any given oplog position is not acknowledged until all messages under that offset are delivered.
+
+*Type*: `int`
+
+*Default*: `1000`
+
+
+=== `read_batch_size`
+
+The number of documents to fetch in each message batch from MongoDB.
+
+
+*Type*: `int`
+
+*Default*: `1000`
+
+
+=== `read_max_wait`
+
+The maximum duration MongoDB waits to accumulate the <<read_batch_size,`read_batch_size`>> documents on a change stream before returning the batch to Redpanda Connect.
+
+*Type*: `string`
+
+*Default*: `1s`
+
+=== `stream_snapshot`
+
+When set to `true`, this input streams a snapshot of all existing data in the source collections before streaming data changes.
+
+*Type*: `bool`
+
+*Default*: `false`
+
+```yml
+# Examples
+
+stream_snapshot: true
+```
+
+=== `snapshot_parallelism`
+
+Specifies the number of connections to use when reading the initial snapshot from one or more collections. Increase this number to enable parallel processing of the snapshot. 
+
+This feature uses the `$splitVector` command to split snapshot data into chunks for more efficient processing.
+
+This field is only applicable when `stream_snapshot` is set to `true`. 
+
+*Type*: `int`
+
+*Default*: `1`
+
+=== `snapshot_auto_bucket_sharding`
+
+Uses the https://www.mongodb.com/docs/manual/reference/operator/aggregation/bucketAuto/[`$bucketAuto`^] command instead of the default, `$splitVector`, to split the snapshot data into chunks for processing. This is required for environments, such as MongoDB Atlas, where privileged access is restricted. To enable parallel processing in these environments:
path: modules/components/pages/inputs/mongodb_cdc.adoc
user: login: rockwotj
body: I think saying that `$splitVector` is a privileged command might make people think it's a scary command :)
created_at: 2025-03-07T15:58:11Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,392 @@
+= mongodb_cdc
+// tag::single-source[]
+:type: input
+:page-beta: true
+:categories: ["Services"]
+
+component_type_dropdown::[]
+
+Streams data changes from a MongoDB replica set, using MongoDB's https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data updates.
+
+ifndef::env-cloud[]
+Introduced in version 4.48.1.
+endif::[]
+
+[tabs]
+======
+Common::
++
+--
+
+```yml
+# Configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: "" # Optional
+    password: "" # Optional
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    auto_replay_nacks: true
+```
+
+--
+Advanced::
++
+--
+
+```yml
+# All configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: ""
+    password: ""
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    snapshot_auto_bucket_sharding: false
+    document_mode: update_lookup
+    json_marshal_mode: canonical
+    app_name: benthos
+    auto_replay_nacks: true
+```
+--
+======
+
+== Prerequisites
+
+- MongoDB version 6 or later
+ifndef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment.
+endif::[]
+ifdef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment. For detailed networking information, including how to set up a VPC peering connection, see xref:networking:index.adoc[Redpanda Cloud Networking].
+endif::[]
+- A MongoDB database running as a https://www.mongodb.com/docs/manual/replication/#replication-in-mongodb[replica set^] or in a https://www.mongodb.com/docs/manual/sharding/[sharded cluster^] using replica set https://www.mongodb.com/docs/manual/reference/replica-configuration/#rsconf.protocolVersion[protocol version 1].
+- A MongoDB database using the https://www.mongodb.com/docs/manual/core/wiredtiger/#storage-wiredtiger[WiredTiger] storage engine.
+
+
+ifndef::env-cloud[]
+== Enable connectivity from cloud-based data sources
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the IP addresses of your Redpanda Connect instances to your firewall rules.
+
+endif::[]
+
+ifdef::env-cloud[]
+
+== Enable connectivity from cloud-based data sources (BYOC)
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the NAT Gateway IP address of your Redpanda cluster to the allowlist of your data source.
+
+endif::[]
+
+== Data capture method
+
+The `mongodb_cdc` input uses https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data changes, which does not propagate _all_ changes to Redpanda Connect. To capture all changes in a MongoDB cluster, including deletions, enable pre- and post-image saving for the cluster and <<collections,required collections>>. For more information, see <<document_mode,`document_mode` options>> and the https://www.mongodb.com/docs/manual/changeStreams/#change-streams-with-document-pre--and-post-images[MongoDB documentation].
+
+
+== Data replication
+
+Redpanda Connect allows you to specify which <<collections,database collections>> in your source database to receive changes from.
+
+You can also run the `mongodb_cdc` input in one of two modes, depending on whether you need a snapshot of existing data before streaming updates.
+
+- Snapshot mode: Redpanda Connect first captures a snapshot of all data in the selected collections and streams the contents before processing changes from the last recorded https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^] position.
+- Streaming mode: Redpanda Connect skips the snapshot and processes only the most recent data changes, starting from the latest oplog position.
+
+=== Snapshot mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `true`, Redpanda Connect connects to your MongoDB database and does the following to capture a snapshot of all data in the selected collections:
+
+. Records the latest oplog position.
+. Determines the strategy for splitting the snapshot data down into shards or chunks for more efficient processing:
+.. If <<snapshot_auto_bucket_sharding,`snapshot_auto_bucket_sharding`>> is set to `false`, the internal `$splitVector` command is used to compute shards.  
+.. If <<snapshot_auto_bucket_sharding,`snapshot_auto_bucket_sharding`>> is set to `true`, the https://www.mongodb.com/docs/manual/reference/operator/aggregation/bucketAuto/[`$bucketAuto`^] command is used instead. This setting is for environments, such as MongoDB Atlas, where privileged access is restricted.
+. This input then uses the number of connections specified in <<snapshot_parallelism,`snapshot-parallelism`>> to read the selected collections.
++
+NOTE: If the pipeline restarts during this process, Redpanda Connect must start the snapshot capture from scratch to store the current oplog position in the <<checkpoint_cache,`checkpoint_cache`>>.
+. Finally, the input uses the stored oplog position to catch up with changes that occurred during snapshot processing.
+
+=== Streaming mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `false`, Redpanda Connect connects to your MongoDB database and starts processing data changes from the latest oplog position. If the pipeline restarts, Redpanda Connect resumes processing updates from the last oplog position written to the <<checkpoint_cache,`checkpoint_cache`>>.
+
+== Metadata
+
+This input adds the following metadata fields to each message:
+
+- `operation`: The type of data change that generated the message: `read`, `create`, `update`, `replace`, `delete`, `update`. A `read` operation occurs when the initial snapshot of the database is processed.
+- `collection`: The name of the collection from which the message originated.
+- `operation_time`: The time the data change was written to the https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^] in the form of a Binary JSON (BSON) timestamp: `{"t": <unix seconds>, "i": <sequence number>}`.
+
+== Fields
+
+=== `url`
+
+The URL of the target MongoDB server.
+
+*Type*: `string`
+
+*Default*: `""`
+
+```yml
+# Examples
+
+url: mongodb://localhost:27017
+```
+
+=== `database`
+
+The name of the MongoDB database to stream changes from.
+
+*Type*: `string`
+
+*Default*: `""`
+
+
+=== `username`
+
+The username to connect to the database.
+
+*Type*: `string`
+
+*Default*: `""`
+
+=== `password`
+
+The password to connect to the database.
+
+include::components:partial$secret_warning.adoc[]
+
+*Type*: `string`
+
+*Default*: `""`
+
+
+=== `collections`
+
+A list of collections to stream changes from. Specify each collection name as a separate item.
+
+*Type*: `array`
+
+*Default*: `[]`
+
+```yml
+# Examples
+
+collections:
+  - orders
+  - customers
+  - inventory
+```
+
+=== `checkpoint_cache`
+
+Specify a `cache` resource to store the oplog position for the most recent data update streamed to Redpanda Connect. After a restart, Redpanda Connect can continue processing changes from this position, avoiding the need to reprocess all collection updates.
path: modules/components/pages/inputs/mongodb_cdc.adoc
user: login: asimms41
body: Updated.
created_at: 2025-03-07T16:35:10Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,392 @@
+= mongodb_cdc
+// tag::single-source[]
+:type: input
+:page-beta: true
+:categories: ["Services"]
+
+component_type_dropdown::[]
+
+Streams data changes from a MongoDB replica set, using MongoDB's https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data updates.
+
+ifndef::env-cloud[]
+Introduced in version 4.48.1.
+endif::[]
+
+[tabs]
+======
+Common::
++
+--
+
+```yml
+# Configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: "" # Optional
+    password: "" # Optional
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    auto_replay_nacks: true
+```
+
+--
+Advanced::
++
+--
+
+```yml
+# All configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: ""
+    password: ""
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    snapshot_auto_bucket_sharding: false
+    document_mode: update_lookup
+    json_marshal_mode: canonical
+    app_name: benthos
+    auto_replay_nacks: true
+```
+--
+======
+
+== Prerequisites
+
+- MongoDB version 6 or later
+ifndef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment.
+endif::[]
+ifdef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment. For detailed networking information, including how to set up a VPC peering connection, see xref:networking:index.adoc[Redpanda Cloud Networking].
+endif::[]
+- A MongoDB database running as a https://www.mongodb.com/docs/manual/replication/#replication-in-mongodb[replica set^] or in a https://www.mongodb.com/docs/manual/sharding/[sharded cluster^] using replica set https://www.mongodb.com/docs/manual/reference/replica-configuration/#rsconf.protocolVersion[protocol version 1].
+- A MongoDB database using the https://www.mongodb.com/docs/manual/core/wiredtiger/#storage-wiredtiger[WiredTiger] storage engine.
+
+
+ifndef::env-cloud[]
+== Enable connectivity from cloud-based data sources
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the IP addresses of your Redpanda Connect instances to your firewall rules.
+
+endif::[]
+
+ifdef::env-cloud[]
+
+== Enable connectivity from cloud-based data sources (BYOC)
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the NAT Gateway IP address of your Redpanda cluster to the allowlist of your data source.
+
+endif::[]
+
+== Data capture method
+
+The `mongodb_cdc` input uses https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data changes, which does not propagate _all_ changes to Redpanda Connect. To capture all changes in a MongoDB cluster, including deletions, enable pre- and post-image saving for the cluster and <<collections,required collections>>. For more information, see <<document_mode,`document_mode` options>> and the https://www.mongodb.com/docs/manual/changeStreams/#change-streams-with-document-pre--and-post-images[MongoDB documentation].
+
+
+== Data replication
+
+Redpanda Connect allows you to specify which <<collections,database collections>> in your source database to receive changes from.
+
+You can also run the `mongodb_cdc` input in one of two modes, depending on whether you need a snapshot of existing data before streaming updates.
+
+- Snapshot mode: Redpanda Connect first captures a snapshot of all data in the selected collections and streams the contents before processing changes from the last recorded https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^] position.
+- Streaming mode: Redpanda Connect skips the snapshot and processes only the most recent data changes, starting from the latest oplog position.
+
+=== Snapshot mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `true`, Redpanda Connect connects to your MongoDB database and does the following to capture a snapshot of all data in the selected collections:
+
+. Records the latest oplog position.
+. Determines the strategy for splitting the snapshot data down into shards or chunks for more efficient processing:
+.. If <<snapshot_auto_bucket_sharding,`snapshot_auto_bucket_sharding`>> is set to `false`, the internal `$splitVector` command is used to compute shards.  
+.. If <<snapshot_auto_bucket_sharding,`snapshot_auto_bucket_sharding`>> is set to `true`, the https://www.mongodb.com/docs/manual/reference/operator/aggregation/bucketAuto/[`$bucketAuto`^] command is used instead. This setting is for environments, such as MongoDB Atlas, where privileged access is restricted.
+. This input then uses the number of connections specified in <<snapshot_parallelism,`snapshot-parallelism`>> to read the selected collections.
++
+NOTE: If the pipeline restarts during this process, Redpanda Connect must start the snapshot capture from scratch to store the current oplog position in the <<checkpoint_cache,`checkpoint_cache`>>.
+. Finally, the input uses the stored oplog position to catch up with changes that occurred during snapshot processing.
+
+=== Streaming mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `false`, Redpanda Connect connects to your MongoDB database and starts processing data changes from the latest oplog position. If the pipeline restarts, Redpanda Connect resumes processing updates from the last oplog position written to the <<checkpoint_cache,`checkpoint_cache`>>.
+
+== Metadata
+
+This input adds the following metadata fields to each message:
+
+- `operation`: The type of data change that generated the message: `read`, `create`, `update`, `replace`, `delete`, `update`. A `read` operation occurs when the initial snapshot of the database is processed.
+- `collection`: The name of the collection from which the message originated.
+- `operation_time`: The time the data change was written to the https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^] in the form of a Binary JSON (BSON) timestamp: `{"t": <unix seconds>, "i": <sequence number>}`.
+
+== Fields
+
+=== `url`
+
+The URL of the target MongoDB server.
+
+*Type*: `string`
+
+*Default*: `""`
+
+```yml
+# Examples
+
+url: mongodb://localhost:27017
+```
+
+=== `database`
+
+The name of the MongoDB database to stream changes from.
+
+*Type*: `string`
+
+*Default*: `""`
+
+
+=== `username`
+
+The username to connect to the database.
+
+*Type*: `string`
+
+*Default*: `""`
+
+=== `password`
+
+The password to connect to the database.
+
+include::components:partial$secret_warning.adoc[]
+
+*Type*: `string`
+
+*Default*: `""`
+
+
+=== `collections`
+
+A list of collections to stream changes from. Specify each collection name as a separate item.
+
+*Type*: `array`
+
+*Default*: `[]`
+
+```yml
+# Examples
+
+collections:
+  - orders
+  - customers
+  - inventory
+```
+
+=== `checkpoint_cache`
+
+Specify a `cache` resource to store the oplog position for the most recent data update streamed to Redpanda Connect. After a restart, Redpanda Connect can continue processing changes from this position, avoiding the need to reprocess all collection updates.
+
+*Type*: `string`
+
+*Default*: `""`
+
+```yml
+# Examples
+
+input:
+  mongodb_cdc:
+    url: mongodb://localhost:27017
+    collections: [my_collection]
+    checkpoint_cache: "my_cdc_cache"
+cache_resources:
+  - label: "my_cdc_cache"
+    redis:
+      url: redis://:6379
+```
+
+=== `checkpoint_key`
+
+The key identifier used to store the oplog position in <<checkpoint_cache,`checkpoint_cache`>>. If you have multiple `mongodb_cdc` inputs sharing the same cache, you can provide an alternative key.
+
+
+*Type*: `string`
+
+*Default*: `mongodb_cdc_checkpoint`
+
+
+=== `checkpoint_interval`
+
+The interval between writing checkpoints to the cache.
+
+*Type*: `string`
+
+*Default*: `5s`
+
+=== `checkpoint_limit`
+
+The maximum number of in-flight messages emitted from this input. Increasing this limit enables parallel processing, and batching at the output level. To preserve at-least-once guarantees, any given oplog position is not acknowledged until all messages under that offset are delivered.
+
+*Type*: `int`
+
+*Default*: `1000`
+
+
+=== `read_batch_size`
+
+The number of documents to fetch in each message batch from MongoDB.
+
+
+*Type*: `int`
+
+*Default*: `1000`
+
+
+=== `read_max_wait`
+
+The maximum duration MongoDB waits to accumulate the <<read_batch_size,`read_batch_size`>> documents on a change stream before returning the batch to Redpanda Connect.
+
+*Type*: `string`
+
+*Default*: `1s`
+
+=== `stream_snapshot`
+
+When set to `true`, this input streams a snapshot of all existing data in the source collections before streaming data changes.
+
+*Type*: `bool`
+
+*Default*: `false`
+
+```yml
+# Examples
+
+stream_snapshot: true
+```
+
+=== `snapshot_parallelism`
+
+Specifies the number of connections to use when reading the initial snapshot from one or more collections. Increase this number to enable parallel processing of the snapshot. 
+
+This feature uses the `$splitVector` command to split snapshot data into chunks for more efficient processing.
+
+This field is only applicable when `stream_snapshot` is set to `true`. 
+
+*Type*: `int`
+
+*Default*: `1`
+
+=== `snapshot_auto_bucket_sharding`
+
+Uses the https://www.mongodb.com/docs/manual/reference/operator/aggregation/bucketAuto/[`$bucketAuto`^] command instead of the default, `$splitVector`, to split the snapshot data into chunks for processing. This is required for environments, such as MongoDB Atlas, where privileged access is restricted. To enable parallel processing in these environments:
path: modules/components/pages/inputs/mongodb_cdc.adoc
user: login: asimms41
body: Updated.
created_at: 2025-03-07T16:35:22Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,392 @@
+= mongodb_cdc
+// tag::single-source[]
+:type: input
+:page-beta: true
+:categories: ["Services"]
+
+component_type_dropdown::[]
+
+Streams data changes from a MongoDB replica set, using MongoDB's https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data updates.
+
+ifndef::env-cloud[]
+Introduced in version 4.48.1.
+endif::[]
+
+[tabs]
+======
+Common::
++
+--
+
+```yml
+# Configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: "" # Optional
+    password: "" # Optional
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    auto_replay_nacks: true
+```
+
+--
+Advanced::
++
+--
+
+```yml
+# All configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: ""
+    password: ""
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    snapshot_auto_bucket_sharding: false
+    document_mode: update_lookup
+    json_marshal_mode: canonical
+    app_name: benthos
+    auto_replay_nacks: true
+```
+--
+======
+
+== Prerequisites
+
+- MongoDB version 6 or later
+ifndef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment.
+endif::[]
+ifdef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment. For detailed networking information, including how to set up a VPC peering connection, see xref:networking:index.adoc[Redpanda Cloud Networking].
+endif::[]
+- A MongoDB database running as a https://www.mongodb.com/docs/manual/replication/#replication-in-mongodb[replica set^] or in a https://www.mongodb.com/docs/manual/sharding/[sharded cluster^] using replica set https://www.mongodb.com/docs/manual/reference/replica-configuration/#rsconf.protocolVersion[protocol version 1].
+- A MongoDB database using the https://www.mongodb.com/docs/manual/core/wiredtiger/#storage-wiredtiger[WiredTiger] storage engine.
+
+
+ifndef::env-cloud[]
+== Enable connectivity from cloud-based data sources
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the IP addresses of your Redpanda Connect instances to your firewall rules.
+
+endif::[]
+
+ifdef::env-cloud[]
+
+== Enable connectivity from cloud-based data sources (BYOC)
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the NAT Gateway IP address of your Redpanda cluster to the allowlist of your data source.
+
+endif::[]
+
+== Data capture method
+
+The `mongodb_cdc` input uses https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data changes, which does not propagate _all_ changes to Redpanda Connect. To capture all changes in a MongoDB cluster, including deletions, enable pre- and post-image saving for the cluster and <<collections,required collections>>. For more information, see <<document_mode,`document_mode` options>> and the https://www.mongodb.com/docs/manual/changeStreams/#change-streams-with-document-pre--and-post-images[MongoDB documentation].
+
+
+== Data replication
+
+Redpanda Connect allows you to specify which <<collections,database collections>> in your source database to receive changes from.
+
+You can also run the `mongodb_cdc` input in one of two modes, depending on whether you need a snapshot of existing data before streaming updates.
+
+- Snapshot mode: Redpanda Connect first captures a snapshot of all data in the selected collections and streams the contents before processing changes from the last recorded https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^] position.
+- Streaming mode: Redpanda Connect skips the snapshot and processes only the most recent data changes, starting from the latest oplog position.
+
+=== Snapshot mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `true`, Redpanda Connect connects to your MongoDB database and does the following to capture a snapshot of all data in the selected collections:
+
+. Records the latest oplog position.
+. Determines the strategy for splitting the snapshot data down into shards or chunks for more efficient processing:
+.. If <<snapshot_auto_bucket_sharding,`snapshot_auto_bucket_sharding`>> is set to `false`, the internal `$splitVector` command is used to compute shards.  
+.. If <<snapshot_auto_bucket_sharding,`snapshot_auto_bucket_sharding`>> is set to `true`, the https://www.mongodb.com/docs/manual/reference/operator/aggregation/bucketAuto/[`$bucketAuto`^] command is used instead. This setting is for environments, such as MongoDB Atlas, where the `$splitVector` command is not available.
+. This input then uses the number of connections specified in <<snapshot_parallelism,`snapshot-parallelism`>> to read the selected collections.
++
+NOTE: If the pipeline restarts during this process, Redpanda Connect must start the snapshot capture from scratch to store the current oplog position in the <<checkpoint_cache,`checkpoint_cache`>>.
+. Finally, the input uses the stored oplog position to catch up with changes that occurred during snapshot processing.
+
+=== Streaming mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `false`, Redpanda Connect connects to your MongoDB database and starts processing data changes from the latest oplog position. If the pipeline restarts, Redpanda Connect resumes processing updates from the last oplog position written to the <<checkpoint_cache,`checkpoint_cache`>>.
+
+== Metadata
+
+This input adds the following metadata fields to each message:
+
+- `operation`: The type of data change that generated the message: `read`, `create`, `update`, `replace`, `delete`, `update`. A `read` operation occurs when the initial snapshot of the database is processed.
+- `collection`: The name of the collection from which the message originated.
+- `operation_time`: The time the data change was written to the https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^] in the form of a Binary JSON (BSON) timestamp: `{"t": <unix seconds>, "i": <sequence number>}`.
+
+== Fields
+
+=== `url`
+
+The URL of the target MongoDB server.
+
+*Type*: `string`
+
+*Default*: `""`
+
+```yml
+# Examples
+
+url: mongodb://localhost:27017
+```
+
+=== `database`
+
+The name of the MongoDB database to stream changes from.
+
+*Type*: `string`
+
+*Default*: `""`
+
+
+=== `username`
+
+The username to connect to the database.
+
+*Type*: `string`
+
+*Default*: `""`
+
+=== `password`
+
+The password to connect to the database.
+
+include::components:partial$secret_warning.adoc[]
+
+*Type*: `string`
+
+*Default*: `""`
+
+
+=== `collections`
+
+A list of collections to stream changes from. Specify each collection name as a separate item.
+
+*Type*: `array`
+
+*Default*: `[]`
+
+```yml
+# Examples
+
+collections:
+  - orders
+  - customers
+  - inventory
+```
+
+=== `checkpoint_cache`
+
+Specify a xref:components:caches/about.adoc[`cache` resource] to store the oplog position for the most recent data update streamed to Redpanda Connect. After a restart, Redpanda Connect can continue processing changes from this position, avoiding the need to reprocess all collection updates.
+
+*Type*: `string`
+
+*Default*: `""`
+
+```yml
+# Examples
+
+input:
+  mongodb_cdc:
+    url: mongodb://localhost:27017
+    collections: [my_collection]
+    checkpoint_cache: "my_cdc_cache"
+cache_resources:
+  - label: "my_cdc_cache"
+    redis:
+      url: redis://:6379
+```
+
+=== `checkpoint_key`
+
+The key identifier used to store the oplog position in <<checkpoint_cache,`checkpoint_cache`>>. If you have multiple `mongodb_cdc` inputs sharing the same cache, you can provide an alternative key.
+
+
+*Type*: `string`
+
+*Default*: `mongodb_cdc_checkpoint`
+
+
+=== `checkpoint_interval`
+
+The interval between writing checkpoints to the cache.
+
+*Type*: `string`
+
+*Default*: `5s`
+
+=== `checkpoint_limit`
+
+The maximum number of in-flight messages emitted from this input. Increasing this limit enables parallel processing, and batching at the output level. To preserve at-least-once guarantees, any given oplog position is not acknowledged until all messages under that offset are delivered.
+
+*Type*: `int`
+
+*Default*: `1000`
+
+
+=== `read_batch_size`
+
+The number of documents to fetch in each message batch from MongoDB.
+
+
+*Type*: `int`
+
+*Default*: `1000`
+
+
+=== `read_max_wait`
+
+The maximum duration MongoDB waits to accumulate the <<read_batch_size,`read_batch_size`>> documents on a change stream before returning the batch to Redpanda Connect.
+
+*Type*: `string`
+
+*Default*: `1s`
+
+=== `stream_snapshot`
+
+When set to `true`, this input streams a snapshot of all existing data in the source collections before streaming data changes.
+
+*Type*: `bool`
+
+*Default*: `false`
+
+```yml
+# Examples
+
+stream_snapshot: true
+```
+
+=== `snapshot_parallelism`
+
+Specifies the number of connections to use when reading the initial snapshot from one or more collections. Increase this number to enable parallel processing of the snapshot. 
+
+This feature uses the `$splitVector` command to split snapshot data into chunks for more efficient processing.
+
+This field is only applicable when `stream_snapshot` is set to `true`. 
+
+*Type*: `int`
+
+*Default*: `1`
+
+=== `snapshot_auto_bucket_sharding`
+
+Uses the https://www.mongodb.com/docs/manual/reference/operator/aggregation/bucketAuto/[`$bucketAuto`^] command instead of the default, `$splitVector`, to split the snapshot data into chunks for processing. This is required for environments, such as MongoDB Atlas, where the `$splitVector` command is not available. To enable parallel processing in these environments:
+
+- Set this field to to `true`.
+- Set `stream_snapshot` to `true`.
+- Increase `snapshot_parallelism` to a value greater than `1`.
+
+*Type*: `bool`
+
+*Default*: `false`
+
+=== `document_mode`
+
+The mode in which MongoDB emits document changes to Redpanda Connect, specifically updates and deletes.
+
+*Type*: `string`
+
+*Default*: `update_lookup`
+
+[cols="1,2"]
+|===
+| Option | Description the document mode
path: modules/components/pages/inputs/mongodb_cdc.adoc
user: login: JakeSCahill
body: ```suggestion
| Option | Description
```
created_at: 2025-03-07T16:37:08Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,392 @@
+= mongodb_cdc
+// tag::single-source[]
+:type: input
+:page-beta: true
path: modules/components/pages/inputs/mongodb_cdc.adoc
user: login: JakeSCahill
body: Is this going to be beta in both Cloud and Self-Managed?
created_at: 2025-03-07T16:37:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,392 @@
+= mongodb_cdc
+// tag::single-source[]
+:type: input
+:page-beta: true
path: modules/components/pages/inputs/mongodb_cdc.adoc
user: login: asimms41
body: It's still labelled as experimental so my understanding is yes. @jrkinley, can you confirm?
created_at: 2025-03-07T16:44:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,392 @@
+= mongodb_cdc
+// tag::single-source[]
+:type: input
+:page-beta: true
+:categories: ["Services"]
+
+component_type_dropdown::[]
+
+Streams data changes from a MongoDB replica set, using MongoDB's https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data updates.
+
+ifndef::env-cloud[]
+Introduced in version 4.48.1.
+endif::[]
+
+[tabs]
+======
+Common::
++
+--
+
+```yml
+# Configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: "" # Optional
+    password: "" # Optional
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    auto_replay_nacks: true
+```
+
+--
+Advanced::
++
+--
+
+```yml
+# All configuration fields, showing default values
+input:
+  label: ""
+  mongodb_cdc:
+    url: mongodb://localhost:27017 # No default (required)
+    database: "" # No default (required)
+    username: ""
+    password: ""
+    collections: [] # No default (required)
+    checkpoint_cache: "" # No default (required)
+    checkpoint_key: mongodb_cdc_checkpoint
+    checkpoint_interval: 5s
+    checkpoint_limit: 1000
+    read_batch_size: 1000
+    read_max_wait: 1s
+    stream_snapshot: false
+    snapshot_parallelism: 1
+    snapshot_auto_bucket_sharding: false
+    document_mode: update_lookup
+    json_marshal_mode: canonical
+    app_name: benthos
+    auto_replay_nacks: true
+```
+--
+======
+
+== Prerequisites
+
+- MongoDB version 6 or later
+ifndef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment.
+endif::[]
+ifdef::env-cloud[]
+- Network access from the cluster where your Redpanda Connect pipeline is running to the source database environment. For detailed networking information, including how to set up a VPC peering connection, see xref:networking:index.adoc[Redpanda Cloud Networking].
+endif::[]
+- A MongoDB database running as a https://www.mongodb.com/docs/manual/replication/#replication-in-mongodb[replica set^] or in a https://www.mongodb.com/docs/manual/sharding/[sharded cluster^] using replica set https://www.mongodb.com/docs/manual/reference/replica-configuration/#rsconf.protocolVersion[protocol version 1].
+- A MongoDB database using the https://www.mongodb.com/docs/manual/core/wiredtiger/#storage-wiredtiger[WiredTiger] storage engine.
+
+
+ifndef::env-cloud[]
+== Enable connectivity from cloud-based data sources
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the IP addresses of your Redpanda Connect instances to your firewall rules.
+
+endif::[]
+
+ifdef::env-cloud[]
+
+== Enable connectivity from cloud-based data sources (BYOC)
+
+To establish a secure connection between a cloud-based data source and Redpanda Connect, you must add the NAT Gateway IP address of your Redpanda cluster to the allowlist of your data source.
+
+endif::[]
+
+== Data capture method
+
+The `mongodb_cdc` input uses https://www.mongodb.com/docs/manual/changeStreams/[change streams^] to capture data changes, which does not propagate _all_ changes to Redpanda Connect. To capture all changes in a MongoDB cluster, including deletions, enable pre- and post-image saving for the cluster and <<collections,required collections>>. For more information, see <<document_mode,`document_mode` options>> and the https://www.mongodb.com/docs/manual/changeStreams/#change-streams-with-document-pre--and-post-images[MongoDB documentation].
+
+
+== Data replication
+
+Redpanda Connect allows you to specify which <<collections,database collections>> in your source database to receive changes from.
+
+You can also run the `mongodb_cdc` input in one of two modes, depending on whether you need a snapshot of existing data before streaming updates.
+
+- Snapshot mode: Redpanda Connect first captures a snapshot of all data in the selected collections and streams the contents before processing changes from the last recorded https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^] position.
+- Streaming mode: Redpanda Connect skips the snapshot and processes only the most recent data changes, starting from the latest oplog position.
+
+=== Snapshot mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `true`, Redpanda Connect connects to your MongoDB database and does the following to capture a snapshot of all data in the selected collections:
+
+. Records the latest oplog position.
+. Determines the strategy for splitting the snapshot data down into shards or chunks for more efficient processing:
+.. If <<snapshot_auto_bucket_sharding,`snapshot_auto_bucket_sharding`>> is set to `false`, the internal `$splitVector` command is used to compute shards.  
+.. If <<snapshot_auto_bucket_sharding,`snapshot_auto_bucket_sharding`>> is set to `true`, the https://www.mongodb.com/docs/manual/reference/operator/aggregation/bucketAuto/[`$bucketAuto`^] command is used instead. This setting is for environments, such as MongoDB Atlas, where the `$splitVector` command is not available.
+. This input then uses the number of connections specified in <<snapshot_parallelism,`snapshot-parallelism`>> to read the selected collections.
++
+NOTE: If the pipeline restarts during this process, Redpanda Connect must start the snapshot capture from scratch to store the current oplog position in the <<checkpoint_cache,`checkpoint_cache`>>.
+. Finally, the input uses the stored oplog position to catch up with changes that occurred during snapshot processing.
+
+=== Streaming mode
+
+If you set the <<stream_snapshot,`stream_snapshot` field>> to `false`, Redpanda Connect connects to your MongoDB database and starts processing data changes from the latest oplog position. If the pipeline restarts, Redpanda Connect resumes processing updates from the last oplog position written to the <<checkpoint_cache,`checkpoint_cache`>>.
+
+== Metadata
+
+This input adds the following metadata fields to each message:
+
+- `operation`: The type of data change that generated the message: `read`, `create`, `update`, `replace`, `delete`, `update`. A `read` operation occurs when the initial snapshot of the database is processed.
+- `collection`: The name of the collection from which the message originated.
+- `operation_time`: The time the data change was written to the https://www.mongodb.com/docs/manual/core/replica-set-oplog/[operations log (oplog)^] in the form of a Binary JSON (BSON) timestamp: `{"t": <unix seconds>, "i": <sequence number>}`.
+
+== Fields
+
+=== `url`
+
+The URL of the target MongoDB server.
+
+*Type*: `string`
+
+*Default*: `""`
+
+```yml
+# Examples
+
+url: mongodb://localhost:27017
+```
+
+=== `database`
+
+The name of the MongoDB database to stream changes from.
+
+*Type*: `string`
+
+*Default*: `""`
+
+
+=== `username`
+
+The username to connect to the database.
+
+*Type*: `string`
+
+*Default*: `""`
+
+=== `password`
+
+The password to connect to the database.
+
+include::components:partial$secret_warning.adoc[]
+
+*Type*: `string`
+
+*Default*: `""`
+
+
+=== `collections`
+
+A list of collections to stream changes from. Specify each collection name as a separate item.
+
+*Type*: `array`
+
+*Default*: `[]`
+
+```yml
+# Examples
+
+collections:
+  - orders
+  - customers
+  - inventory
+```
+
+=== `checkpoint_cache`
+
+Specify a xref:components:caches/about.adoc[`cache` resource] to store the oplog position for the most recent data update streamed to Redpanda Connect. After a restart, Redpanda Connect can continue processing changes from this position, avoiding the need to reprocess all collection updates.
+
+*Type*: `string`
+
+*Default*: `""`
+
+```yml
+# Examples
+
+input:
+  mongodb_cdc:
+    url: mongodb://localhost:27017
+    collections: [my_collection]
+    checkpoint_cache: "my_cdc_cache"
+cache_resources:
+  - label: "my_cdc_cache"
+    redis:
+      url: redis://:6379
+```
+
+=== `checkpoint_key`
+
+The key identifier used to store the oplog position in <<checkpoint_cache,`checkpoint_cache`>>. If you have multiple `mongodb_cdc` inputs sharing the same cache, you can provide an alternative key.
+
+
+*Type*: `string`
+
+*Default*: `mongodb_cdc_checkpoint`
+
+
+=== `checkpoint_interval`
+
+The interval between writing checkpoints to the cache.
+
+*Type*: `string`
+
+*Default*: `5s`
+
+=== `checkpoint_limit`
+
+The maximum number of in-flight messages emitted from this input. Increasing this limit enables parallel processing, and batching at the output level. To preserve at-least-once guarantees, any given oplog position is not acknowledged until all messages under that offset are delivered.
+
+*Type*: `int`
+
+*Default*: `1000`
+
+
+=== `read_batch_size`
+
+The number of documents to fetch in each message batch from MongoDB.
+
+
+*Type*: `int`
+
+*Default*: `1000`
+
+
+=== `read_max_wait`
+
+The maximum duration MongoDB waits to accumulate the <<read_batch_size,`read_batch_size`>> documents on a change stream before returning the batch to Redpanda Connect.
+
+*Type*: `string`
+
+*Default*: `1s`
+
+=== `stream_snapshot`
+
+When set to `true`, this input streams a snapshot of all existing data in the source collections before streaming data changes.
+
+*Type*: `bool`
+
+*Default*: `false`
+
+```yml
+# Examples
+
+stream_snapshot: true
+```
+
+=== `snapshot_parallelism`
+
+Specifies the number of connections to use when reading the initial snapshot from one or more collections. Increase this number to enable parallel processing of the snapshot. 
+
+This feature uses the `$splitVector` command to split snapshot data into chunks for more efficient processing.
+
+This field is only applicable when `stream_snapshot` is set to `true`. 
+
+*Type*: `int`
+
+*Default*: `1`
+
+=== `snapshot_auto_bucket_sharding`
+
+Uses the https://www.mongodb.com/docs/manual/reference/operator/aggregation/bucketAuto/[`$bucketAuto`^] command instead of the default, `$splitVector`, to split the snapshot data into chunks for processing. This is required for environments, such as MongoDB Atlas, where the `$splitVector` command is not available. To enable parallel processing in these environments:
+
+- Set this field to to `true`.
+- Set `stream_snapshot` to `true`.
+- Increase `snapshot_parallelism` to a value greater than `1`.
+
+*Type*: `bool`
+
+*Default*: `false`
+
+=== `document_mode`
+
+The mode in which MongoDB emits document changes to Redpanda Connect, specifically updates and deletes.
+
+*Type*: `string`
+
+*Default*: `update_lookup`
+
+[cols="1,2"]
+|===
+| Option | Description the document mode
path: modules/components/pages/inputs/mongodb_cdc.adoc
user: login: asimms41
body: Updated.
created_at: 2025-03-07T16:46:27Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,392 @@
+= mongodb_cdc
+// tag::single-source[]
+:type: input
+:page-beta: true
path: modules/components/pages/inputs/mongodb_cdc.adoc
user: login: JakeSCahill
body: Since you already define it separately in the Cloud PR, I'd move this out of the single-source tag so we can remove beta individually for Cloud/Self-Managed: modules/develop/pages/connect/components/inputs/mongodb_cdc.adoc
created_at: 2025-03-07T16:56:54Z
author_association: CONTRIBUTOR

PR comments:
user: login: netlify[bot]
created_at: 2025-03-06T17:55:44Z
body: ### <span aria-hidden="true">✅</span> Deploy Preview for *redpanda-connect* ready!


|  Name | Link |
|:-:|------------------------|
|<span aria-hidden="true">🔨</span> Latest commit |  |
|<span aria-hidden="true">🔍</span> Latest deploy log | https://app.netlify.com/sites/redpanda-connect/deploys/67d05a4a7988675d1dc1ca51 |
|<span aria-hidden="true">😎</span> Deploy Preview | [https://deploy-preview-186--redpanda-connect.netlify.app](https://deploy-preview-186--redpanda-connect.netlify.app) |
|<span aria-hidden="true">📱</span> Preview on mobile | <details><summary> Toggle QR Code... </summary><br /><br />![QR Code](https://app.netlify.com/qr-code/eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ1cmwiOiJodHRwczovL2RlcGxveS1wcmV2aWV3LTE4Ni0tcmVkcGFuZGEtY29ubmVjdC5uZXRsaWZ5LmFwcCJ9.4arpdOAdkBGvOj6KUvd1ihmb3qdHaxSBIb2QsBY5ZmU)<br /><br />_Use your smartphone camera to open QR code link._</details> |
---
<!-- [redpanda-connect Preview](https://deploy-preview-186--redpanda-connect.netlify.app) -->
_To edit notification comments on pull requests, go to your [Netlify site configuration](https://app.netlify.com/sites/redpanda-connect/configuration/notifications#deploy-webhooks)._
author_association: NONE
user: login: rockwotj
created_at: 2025-03-07T16:00:35Z
body: Adding @jrkinley to reviews as well going forward so product can have input on the docs if desired.
author_association: CONTRIBUTOR