Repository: meltano/meltano (language: python)

Issue:
feature: Bump the default buffer size
### Feature scope

Configuration (settings parsing, validation, etc.)

### Description

It's currently `10MiB` bytes, but we could probably bump it to `100MiB`:

https://github.com/meltano/meltano/blob/3f7efdfd38c39d3ba928dfc1ff025b1fe1a3e36d/src/meltano/core/bundle/settings.yml#L34-L35

- https://docs.meltano.com/reference/settings/#eltbuffer_size


Discussion / hints:

Fix patch:
diff --git a/docs/docs/reference/settings.mdx b/docs/docs/reference/settings.mdx
index 7c026bbea0..947dcfc4dd 100644
--- a/docs/docs/reference/settings.mdx
+++ b/docs/docs/reference/settings.mdx
@@ -650,7 +650,7 @@ These settings can be used to modify the behavior of [`meltano el`](/reference/c
 ### `elt.buffer_size`
 
 - [Environment variable](/guide/configuration#configuring-settings): `MELTANO_ELT_BUFFER_SIZE`
-- Default: `10485760` (10MiB in bytes)
+- Default: `104857600` (100MiB in bytes)
 
 Size (in bytes) of the buffer between extractor and loader (Singer tap and target) that stores
 [messages](https://hub.meltano.com/singer/spec#messages)
@@ -661,7 +661,7 @@ at which point the extractor will be blocked until the loader has worked through
 of the buffer size available again for new extractor output.
 
 The length of a single line of extractor output is limited to half the buffer size.
-With a default buffer size of 10MiB, the maximum message size would therefore be 5MiB.
+With a default buffer size of 100MiB, the maximum message size would therefore be 50MiB.
 
 #### How to use
 
diff --git a/integration/meltano-manifest/expected-manifests/meltano-manifest.cicd.json b/integration/meltano-manifest/expected-manifests/meltano-manifest.cicd.json
index bb55e81e49..782caf9026 100644
--- a/integration/meltano-manifest/expected-manifests/meltano-manifest.cicd.json
+++ b/integration/meltano-manifest/expected-manifests/meltano-manifest.cicd.json
@@ -50,7 +50,7 @@
     "MELTANO_DATABASE_RETRY_TIMEOUT": "5",
     "MELTANO_DATABASE_URI": "sqlite:///${MELTANO_PROJECT_ROOT}/.meltano/meltano.db",
     "MELTANO_DEFAULT_ENVIRONMENT": "userdev",
-    "MELTANO_ELT_BUFFER_SIZE": "10485760",
+    "MELTANO_ELT_BUFFER_SIZE": "104857600",
     "MELTANO_EXPERIMENTAL": "false",
     "MELTANO_FF_PLUGIN_LOCKS_REQUIRED": "false",
     "MELTANO_FF_STRICT_ENV_VAR_MODE": "false",
diff --git a/integration/meltano-manifest/expected-manifests/meltano-manifest.jigsaw.json b/integration/meltano-manifest/expected-manifests/meltano-manifest.jigsaw.json
index 736eac91db..1e3275bdf9 100644
--- a/integration/meltano-manifest/expected-manifests/meltano-manifest.jigsaw.json
+++ b/integration/meltano-manifest/expected-manifests/meltano-manifest.jigsaw.json
@@ -35,7 +35,7 @@
     "MELTANO_DATABASE_RETRY_TIMEOUT": "5",
     "MELTANO_DATABASE_URI": "sqlite:///${MELTANO_PROJECT_ROOT}/.meltano/meltano.db",
     "MELTANO_DEFAULT_ENVIRONMENT": "userdev",
-    "MELTANO_ELT_BUFFER_SIZE": "10485760",
+    "MELTANO_ELT_BUFFER_SIZE": "104857600",
     "MELTANO_EXPERIMENTAL": "false",
     "MELTANO_FF_PLUGIN_LOCKS_REQUIRED": "false",
     "MELTANO_FF_STRICT_ENV_VAR_MODE": "false",
diff --git a/integration/meltano-manifest/expected-manifests/meltano-manifest.json b/integration/meltano-manifest/expected-manifests/meltano-manifest.json
index 1ad3e4f9e0..73b8b808a4 100644
--- a/integration/meltano-manifest/expected-manifests/meltano-manifest.json
+++ b/integration/meltano-manifest/expected-manifests/meltano-manifest.json
@@ -30,7 +30,7 @@
     "MELTANO_DATABASE_RETRY_TIMEOUT": "5",
     "MELTANO_DATABASE_URI": "sqlite:///${MELTANO_PROJECT_ROOT}/.meltano/meltano.db",
     "MELTANO_DEFAULT_ENVIRONMENT": "userdev",
-    "MELTANO_ELT_BUFFER_SIZE": "10485760",
+    "MELTANO_ELT_BUFFER_SIZE": "104857600",
     "MELTANO_EXPERIMENTAL": "false",
     "MELTANO_FF_PLUGIN_LOCKS_REQUIRED": "false",
     "MELTANO_FF_STRICT_ENV_VAR_MODE": "false",
diff --git a/integration/meltano-manifest/expected-manifests/meltano-manifest.prod.json b/integration/meltano-manifest/expected-manifests/meltano-manifest.prod.json
index 7f048a6bf5..6a231501fd 100644
--- a/integration/meltano-manifest/expected-manifests/meltano-manifest.prod.json
+++ b/integration/meltano-manifest/expected-manifests/meltano-manifest.prod.json
@@ -31,7 +31,7 @@
     "MELTANO_DATABASE_RETRY_TIMEOUT": "5",
     "MELTANO_DATABASE_URI": "sqlite:///${MELTANO_PROJECT_ROOT}/.meltano/meltano.db",
     "MELTANO_DEFAULT_ENVIRONMENT": "userdev",
-    "MELTANO_ELT_BUFFER_SIZE": "10485760",
+    "MELTANO_ELT_BUFFER_SIZE": "104857600",
     "MELTANO_EXPERIMENTAL": "false",
     "MELTANO_FF_PLUGIN_LOCKS_REQUIRED": "false",
     "MELTANO_FF_STRICT_ENV_VAR_MODE": "false",
diff --git a/integration/meltano-manifest/expected-manifests/meltano-manifest.staging.json b/integration/meltano-manifest/expected-manifests/meltano-manifest.staging.json
index ccfb46983a..cbe838cb26 100644
--- a/integration/meltano-manifest/expected-manifests/meltano-manifest.staging.json
+++ b/integration/meltano-manifest/expected-manifests/meltano-manifest.staging.json
@@ -31,7 +31,7 @@
     "MELTANO_DATABASE_RETRY_TIMEOUT": "5",
     "MELTANO_DATABASE_URI": "sqlite:///${MELTANO_PROJECT_ROOT}/.meltano/meltano.db",
     "MELTANO_DEFAULT_ENVIRONMENT": "userdev",
-    "MELTANO_ELT_BUFFER_SIZE": "10485760",
+    "MELTANO_ELT_BUFFER_SIZE": "104857600",
     "MELTANO_EXPERIMENTAL": "false",
     "MELTANO_FF_PLUGIN_LOCKS_REQUIRED": "false",
     "MELTANO_FF_STRICT_ENV_VAR_MODE": "false",
diff --git a/integration/meltano-manifest/expected-manifests/meltano-manifest.userdev.json b/integration/meltano-manifest/expected-manifests/meltano-manifest.userdev.json
index 78abe2647f..1c1d284c81 100644
--- a/integration/meltano-manifest/expected-manifests/meltano-manifest.userdev.json
+++ b/integration/meltano-manifest/expected-manifests/meltano-manifest.userdev.json
@@ -38,7 +38,7 @@
     "MELTANO_DATABASE_RETRY_TIMEOUT": "5",
     "MELTANO_DATABASE_URI": "sqlite:///${MELTANO_PROJECT_ROOT}/.meltano/meltano.db",
     "MELTANO_DEFAULT_ENVIRONMENT": "userdev",
-    "MELTANO_ELT_BUFFER_SIZE": "10485760",
+    "MELTANO_ELT_BUFFER_SIZE": "104857600",
     "MELTANO_EXPERIMENTAL": "false",
     "MELTANO_FF_PLUGIN_LOCKS_REQUIRED": "false",
     "MELTANO_FF_STRICT_ENV_VAR_MODE": "false",
diff --git a/src/meltano/core/bundle/settings.yml b/src/meltano/core/bundle/settings.yml
index bb067f4a84..e74607197f 100644
--- a/src/meltano/core/bundle/settings.yml
+++ b/src/meltano/core/bundle/settings.yml
@@ -32,7 +32,7 @@ settings:
 - name: hub_url_auth
 - name: elt.buffer_size
   kind: integer
-  value: 10485760 # 10 MiB
+  value: 104_857_600 # 100 MiB
 - name: python
 - name: auto_install
   kind: boolean


Test patch:
diff --git a/tests/meltano/cli/test_elt.py b/tests/meltano/cli/test_elt.py
index 35bb3ba488..bcf96c9d8b 100644
--- a/tests/meltano/cli/test_elt.py
+++ b/tests/meltano/cli/test_elt.py
@@ -780,8 +780,8 @@ async def wait_mock():
                         None,
                         (
                             "The extractor generated a message exceeding the "
-                            "message size limit of 5.0MiB (half the buffer "
-                            "size of 10.0MiB)."
+                            "message size limit of 50.0MiB (half the buffer "
+                            "size of 100.0MiB)."
                         ),
                         "error",
                     ),