Repository: airbytehq/airbyte
Pull request #59716: chore: non-dagger builds.
Author: login: davinchia
Labels: name: connectors/destination/s3
name: connectors/source/mysql
Created: 2025-05-07T22:33:52Z | Merged: 2025-05-09T00:04:22Z

## What
- Introduce the airbyte gradle docker plugin to build JVM connectors.
- Add logic to allow us to slowly migrate from one build set up to the other.

## How
The majority of the work is taking in the metadata.yaml file, and converting that to buildArgs to be injected into the airbyte gradle docker plugin.

In particular, we extract the base image from the metadata file. We also add the connector name by referencing the module. This outputs a file buildArgs.properties into the build folder. This is done by the `DockerGenerateConnectorBuildArgs` task.

The `AirbyteDockerConventionPlugin` then takes in the generated buildArgs.properties file and feeds it into the Platform Docker plugin. It also feeds in the top-level Dockerfile, so the Docker plugin can copy it into the build folder.

![Screenshot 2025-05-07 at 3 53 05 PM](https://github.com/user-attachments/assets/36cb2a12-4393-4549-a94c-ecfcbf7437a0)

After this, my plan is move all certifed sources and destinations onto the new system in a source-PR and a destination-PR.

Then move the remaining connectors over in another PR.

## Review guide
Two main files:
- buildSrc/src/main/groovy/io/airbyte/gradle/DockerGenerateConnectorBuildArgs.groovy - contains the task to read the metadata.yaml file and output the buildArgs.properties file.
- buildSrc/src/main/groovy/airbyte-docker-convention.gradle - consumes the buildArgs.properties files and plugs it into the platform Docker plugin.
- airbyte-integrations/connectors/destination-s3/build.gradle is an example for how to configure this.

## User Impact
<!--
* What is the end result perceived by the user?
* If there are negative side effects, please list them. 
-->

## Can this PR be safely reverted and rolled back?
<!--
* If unsure, leave it blank.
-->
- [x] YES 💚
- [ ] NO ❌



Diff:
diff --git a/airbyte-integrations/connectors/build.gradle b/airbyte-integrations/connectors/build.gradle
index fbcae24858dd..ab16b7e21caf 100644
--- a/airbyte-integrations/connectors/build.gradle
+++ b/airbyte-integrations/connectors/build.gradle
@@ -1,6 +1,7 @@
 plugins {
     id 'base'
     id 'ru.vyarus.use-python' version '2.3.0'
+    id "io.airbyte.gradle.docker" version "0.63.0" apply false
     id 'com.github.eirnym.js2p' version "1.0" apply false
 }
 
@@ -81,8 +82,8 @@ allprojects {
         return task
     }
 
-    // Build connector image as part of 'assemble' task.
-    // This is required for local 'integrationTest' execution.
+    // This allows us to slowly migrate connectors between dagger and the new plugins.
+    // This can be deleted once we no longer use Dagger.
     def buildConnectorImage = airbyteCIConnectorsTask(
             'buildConnectorImage', '--disable-report-auto-open', 'build', '--use-host-gradle-dist-tar')
     buildConnectorImage.configure {
@@ -91,9 +92,14 @@ allprojects {
         // Ensure that all files exist beforehand.
         dependsOn tasks.matching { it.name == 'generate' }
     }
-    tasks.named('assemble').configure {
-        // We may revisit the dependency on assemble but the dependency should always be on a base task.
-        dependsOn buildConnectorImage
+
+    afterEvaluate { Project p ->
+        if (!plugins.hasPlugin('airbyte-connector-docker-convention')) {
+            tasks.named('assemble').configure {
+                // We may revisit the dependency on assemble but the dependency should always be on a base task.
+                dependsOn buildConnectorImage
+            }
+        }
     }
 
     // Convenience tasks for local airbyte-ci execution.
diff --git a/buildSrc/build.gradle b/buildSrc/build.gradle
index 21109a19fa65..50001dcaf11f 100644
--- a/buildSrc/build.gradle
+++ b/buildSrc/build.gradle
@@ -3,10 +3,15 @@ plugins {
 }
 
 repositories {
-    // # Gradle looks for dependency artifacts in repositories listed in 'repositories' blocks in descending order.
+    // Gradle looks for dependency artifacts in repositories listed in 'repositories' blocks in descending order.
     gradlePluginPortal()
 }
 
+dependencies {
+    // Note this dependency version needs to line up with the generateJsonSchema2Pojo plugin used in the project.
+    implementation 'org.yaml:snakeyaml:1.33'
+}
+
 tasks.withType(Jar).configureEach {
     duplicatesStrategy DuplicatesStrategy.INCLUDE
 }
diff --git a/buildSrc/src/main/groovy/airbyte-connector-docker-convention.gradle b/buildSrc/src/main/groovy/airbyte-connector-docker-convention.gradle
new file mode 100644
index 000000000000..f9cfb6f17df9
--- /dev/null
+++ b/buildSrc/src/main/groovy/airbyte-connector-docker-convention.gradle
@@ -0,0 +1,85 @@
+import io.airbyte.gradle.DockerGenerateConnectorBuildArgs
+import org.gradle.api.Plugin
+import org.gradle.api.Project
+import org.gradle.api.Task
+import org.gradle.api.file.RegularFile
+import org.gradle.api.plugins.ExtensionAware
+import org.gradle.api.provider.Provider
+
+class AirbyteConnectorDockerConventionPlugin implements Plugin<Project> {
+    private static final String SHARED_DOCKERFILE = 'docker-images/Dockerfile.java-connector-non-airbyte-ci'
+
+    @Override
+    void apply(Project project) {
+        /*
+            We deliberately avoid importing classes from the published Airbyte Docker
+            plugin. That keeps buildSrc free of a compile‑time dependency on a specific
+            plugin version and lets this convention plugin compile before external
+            plugins are resolved.
+         */
+
+        project.plugins.withId('io.airbyte.gradle.docker') {
+            def dockerExt = ((project.extensions.findByName('airbyte') as ExtensionAware)
+                    ?.extensions?.findByName('docker'))
+
+            if (!dockerExt) {
+                project.logger.warn('airbyte.docker extension not found; skipping convention plugin.')
+                return
+            }
+
+            /* ------------------------------------------------------------
+             * 1. imageName default = project.name (if not already set)
+             * ------------------------------------------------------------ */
+            if (!dockerExt.imageName.present) {
+                dockerExt.imageName.convention(project.name)
+            }
+
+            /* ------------------------------------------------------------
+             * 2. Use shared Dockerfile if not overridden. This allows
+             *    users to specify the Dockerfiles per connector.
+             *    The core plugin’s dockerCopyDockerfile task will copy
+             *    it into build/airbyte/docker/Dockerfile automatically.
+             * ------------------------------------------------------------ */
+            if (!dockerExt.dockerfile.present) {
+                dockerExt.dockerfile.set(project.rootProject.file(SHARED_DOCKERFILE))
+            }
+
+            /* ------------------------------------------------------------
+             * 3. Task: generate buildArgs.properties from metadata.yaml
+             * ------------------------------------------------------------ */
+            def genProps = project.tasks.register(
+                    'generateConnectorDockerBuildArgs',
+                    DockerGenerateConnectorBuildArgs
+            ) {
+                metadata.set(project.layout.projectDirectory.file('metadata.yaml'))
+                output.set(project.layout.buildDirectory.file('docker/buildArgs.properties'))
+            }
+
+            /* ------------------------------------------------------------
+             * 4. Lazy provider: read and inject the file generated by the
+             *    DockerGenerateConnectorBuildArgs task as buildArgs
+             * ------------------------------------------------------------ */
+            def propsMapProvider = project.providers
+                    .fileContents(genProps.flatMap { it.output } as Provider<RegularFile>)
+                    .asText
+                    .map { txt ->
+                        txt.readLines()
+                                .findAll { it.contains('=') }
+                                .collectEntries { ln ->
+                                    def (k, v) = ln.split('=', 2)
+                                    [(k.trim()): v.trim()]
+                                }
+                    }
+            dockerExt.buildArgs.putAll(propsMapProvider)
+
+            /* ------------------------------------------------------------
+             * 5. Make dockerBuildx depend on the generator, and register
+             *    the properties file as an input (incremental build‑cache)
+             * ------------------------------------------------------------ */
+            project.tasks.named('dockerBuildx').configure { Task t ->
+                t.dependsOn(genProps)
+                t.inputs.file(genProps.flatMap { it.output })
+            }
+        }
+    }
+}
diff --git a/buildSrc/src/main/groovy/io/airbyte/gradle/DockerGenerateConnectorBuildArgs.groovy b/buildSrc/src/main/groovy/io/airbyte/gradle/DockerGenerateConnectorBuildArgs.groovy
new file mode 100644
index 000000000000..c0d0e7c42e76
--- /dev/null
+++ b/buildSrc/src/main/groovy/io/airbyte/gradle/DockerGenerateConnectorBuildArgs.groovy
@@ -0,0 +1,51 @@
+package io.airbyte.gradle
+
+import org.gradle.api.DefaultTask
+import org.gradle.api.file.RegularFileProperty
+import org.gradle.api.provider.Property
+import org.gradle.api.tasks.*
+import org.yaml.snakeyaml.Yaml
+
+/**
+ * Reads `metadata.yaml` (if present) and emits
+ * `${buildDir}/docker/buildArgs.properties`
+ * so docker build tasks can consume per‑module arguments.
+ */
+class DockerGenerateConnectorBuildArgs extends DefaultTask {
+
+    @InputFile
+    final RegularFileProperty metadata = project.objects.fileProperty()
+
+    @OutputFile
+    final RegularFileProperty output = project.objects.fileProperty()
+
+    @Internal
+    final Property<Yaml> yaml = project.objects.property(Yaml).convention(new Yaml())
+
+    @TaskAction
+    void run() {
+        def metaFile = metadata.get().asFile
+        def outFile = output.get().asFile
+
+        if (!metaFile.exists()) {
+            outFile.text = ''          // keep Gradle’s cache happy
+            return
+        }
+
+        Map root = yaml.get().load(metaFile.text) ?: [:]
+        Map<String, ?> opts = ((root['data'] ?: [:])['connectorBuildOptions'] ?: [:]) as Map<String, ?>
+        outFile.withPrintWriter { pw ->
+            opts.each { k, v ->
+                if (v != null) {
+                    def key = k
+                            .replaceAll(/([a-z0-9])([A-Z])/, '$1_$2') // camelCase → snake_case
+                            .replace('-', '_')                        // dash → underscore
+                            .toUpperCase(Locale.ROOT)
+                    pw.println("${key}=$v")
+                }
+            }
+            /* Always add CONNECTOR_NAME=<module‑directory>. This is used to defined the image's name. */
+            pw.println("CONNECTOR_NAME=${project.projectDir.name}")
+        }
+    }
+}
diff --git a/docker-images/Dockerfile.java-connector-non-airbyte-ci b/docker-images/Dockerfile.java-connector-non-airbyte-ci
new file mode 100644
index 000000000000..79c2617fde51
--- /dev/null
+++ b/docker-images/Dockerfile.java-connector-non-airbyte-ci
@@ -0,0 +1,30 @@
+# syntax=docker/dockerfile:1
+# check=skip=InvalidDefaultArgInFrom
+
+# Java connector Dockerfile for Airbyte
+
+# Build arguments
+ARG BASE_IMAGE
+
+# Base image - using Amazon Corretto (Amazon's distribution of OpenJDK)
+FROM ${BASE_IMAGE}
+ARG CONNECTOR_NAME
+
+# Set permissions for downloaded files
+RUN chmod +x /airbyte/base.sh /airbyte/javabase.sh && \
+    chown airbyte:airbyte /airbyte/base.sh /airbyte/javabase.sh /airbyte/dd-java-agent.jar
+
+ENV AIRBYTE_ENTRYPOINT="/airbyte/base.sh"
+ENV APPLICATION="${CONNECTOR_NAME}"
+
+# Add the connector TAR file and extract it
+COPY airbyte-app.tar /tmp/${CONNECTOR_NAME}.tar
+RUN tar xf /tmp/${CONNECTOR_NAME}.tar --strip-components=1 -C /airbyte && \
+    rm -rf /tmp/${CONNECTOR_NAME}.tar && \
+    chown -R airbyte:airbyte /airbyte
+
+# Set the non-root user
+USER airbyte
+
+# Set entrypoint
+ENTRYPOINT ["/airbyte/base.sh"]
diff --git a/settings.gradle b/settings.gradle
index 038796f48094..34717eabb196 100644
--- a/settings.gradle
+++ b/settings.gradle
@@ -5,7 +5,9 @@ pluginManagement {
         // # Gradle looks for dependency artifacts in repositories listed in 'repositories' blocks in descending order.
 
         // ## Prefer repos controlled by Airbyte.
-        // TODO: add airbyte-controlled proxy repos here
+        maven {
+            url = 'https://airbyte.mycloudrepo.io/public/repositories/airbyte-public-jars/'
+        }
 
         // ## Look into other, public repos.
         // Gradle plugin portal.


Reviews:
user: login: davinchia
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T23:48:22Z
user: login: aaronsteers
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-08T00:12:13Z
user: login: davinchia
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-08T00:14:21Z
user: login: aaronsteers
body: One question regarding re-use/update of existing Dockerfile vs creating a new one. Someone with more Gradle experience may see things I'm missing.

Approving to unblock, and because it looks good from my side. 👍 ✅ 
state: APPROVED
author_association: MEMBER
submitted_at: 2025-05-08T00:18:56Z
user: login: aaronsteers
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-08T00:21:25Z
user: login: aaronsteers
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-08T00:22:42Z
user: login: davinchia
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-08T18:38:16Z
user: login: edgao
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-08T21:16:54Z
user: login: davinchia
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-08T21:30:09Z
user: login: davinchia
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-08T21:43:56Z
user: login: edgao
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-08T23:35:16Z

Review comments:
path: docker-images/Dockerfile.java-connector-non-airbyte-ci
user: login: davinchia
body: Cole gave some feedback that this dockerfile can be optimised by combining some of the commands together.

I'm going to do so in a follow up PR to keep things readable.
created_at: 2025-05-07T23:48:22Z
author_association: CONTRIBUTOR
path: docker-images/Dockerfile.java-connector-non-airbyte-ci
user: login: aaronsteers
body: Any reason not to modify `docker-images/Dockerfile.java-connector` directly? There's no other dependency on that file, and I created it exactly for this purpose. You can safely modify it without worrying about breaking anything in `airbyte-ci`.
created_at: 2025-05-08T00:12:13Z
author_association: MEMBER
path: docker-images/Dockerfile.java-connector-non-airbyte-ci
user: login: davinchia
body: My thought was to leave it there so we have a copy of the existing dagger dockerfile just in case something really bad happens and we need to majorly revert/a reference point to what happens now without reading python code 😅 
created_at: 2025-05-08T00:14:21Z
author_association: CONTRIBUTOR
path: docker-images/Dockerfile.java-connector-non-airbyte-ci
user: login: aaronsteers
body: @davinchia - Except it's not really valuable for that purpose because it's not _exactly_ the same as the Dagger-generated build. I reverse-engineered it as well as I was able - with some small updates. But just to be clear, it isn't really a safe fallback as of now anyway, so do feel free to update. If we want to memorialize it, we can copy-paste the contents into the readme below - since what is pasted there now is already broken and not working anyway according my last review.

https://github.com/airbytehq/airbyte/blob/main/airbyte-ci/connectors/base_images/README.md#L39
created_at: 2025-05-08T00:21:24Z
author_association: MEMBER
path: docker-images/Dockerfile.java-connector-non-airbyte-ci
user: login: aaronsteers
body: To be clear, totally up to you how you want to tackle it. I just want to make sure you know its not really valuable as of now _until/unless_ we test it. 
created_at: 2025-05-08T00:22:42Z
author_association: MEMBER
path: airbyte-integrations/connectors/destination-s3/build.gradle
user: login: davinchia
body: @edgao I confirmed dagger is running the finalize_build.sh script, and this is how the new set up can do so.

This means any connector that wants to install specific binaries has an escape hatch.
created_at: 2025-05-08T18:38:16Z
author_association: CONTRIBUTOR
path: airbyte-integrations/connectors/destination-s3/build.gradle
user: login: edgao
body: checking: this is just copied from the base dockerfile? (I think that's fine, since it means connector devs can optimize the dockerfile, just want to make sure I understand)

if yes: probably worth having a bigger comment saying "this is the only different thing"
created_at: 2025-05-08T21:14:51Z
author_association: CONTRIBUTOR
path: airbyte-integrations/connectors/destination-s3/build.gradle
user: login: davinchia
body: yes, it is copied from the base dockerfile.

will add the comment.
created_at: 2025-05-08T21:30:09Z
author_association: CONTRIBUTOR
path: airbyte-integrations/connectors/destination-s3/build.gradle
user: login: davinchia
body: added
created_at: 2025-05-08T21:43:56Z
author_association: CONTRIBUTOR

PR comments:
user: login: vercel[bot]
created_at: 2025-05-07T22:33:57Z
body: [vc]: #206tLZzIouUDIr0gGXXK8FCWQ1R8GAxN/wP6iOmHU4I=:eyJpc01vbm9yZXBvIjp0cnVlLCJ0eXBlIjoiZ2l0aHViIiwicHJvamVjdHMiOlt7Im5hbWUiOiJhaXJieXRlLWRvY3MiLCJpbnNwZWN0b3JVcmwiOiJodHRwczovL3ZlcmNlbC5jb20vYWlyYnl0ZS1ncm93dGgvYWlyYnl0ZS1kb2NzLzlaNEtWdEhmSkoxM2E1THFTRFM5TWQ5N3lKN2siLCJwcmV2aWV3VXJsIjoiYWlyYnl0ZS1kb2NzLWdpdC1kYXZpbmNoaWEtbm9uLWRhZ2dlci1idWlsZHMtYWlyYnl0ZS1ncm93dGgudmVyY2VsLmFwcCIsIm5leHRDb21taXRTdGF0dXMiOiJERVBMT1lFRCIsImxpdmVGZWVkYmFjayI6eyJyZXNvbHZlZCI6MCwidW5yZXNvbHZlZCI6MCwidG90YWwiOjAsImxpbmsiOiJhaXJieXRlLWRvY3MtZ2l0LWRhdmluY2hpYS1ub24tZGFnZ2VyLWJ1aWxkcy1haXJieXRlLWdyb3d0aC52ZXJjZWwuYXBwIn0sInJvb3REaXJlY3RvcnkiOiJkb2N1c2F1cnVzIn1dfQ==
**The latest updates on your projects**. Learn more about [Vercel for Git ↗︎](https://vercel.link/github-learn-more)

| Name | Status | Preview | Comments | Updated (UTC) |
| :--- | :----- | :------ | :------- | :------ |
| **airbyte-docs** | ✅ Ready ([Inspect](https://vercel.com/airbyte-growth/airbyte-docs/9Z4KVtHfJJ13a5LqSDS9Md97yJ7k)) | [Visit Preview](https://airbyte-docs-git-davinchia-non-dagger-builds-airbyte-growth.vercel.app) | 💬 [**Add feedback**](https://vercel.live/open-feedback/airbyte-docs-git-davinchia-non-dagger-builds-airbyte-growth.vercel.app?via=pr-comment-feedback-link) | May 8, 2025 11:51pm |


author_association: NONE
user: login: github-actions[bot]
created_at: 2025-05-07T22:52:18Z
body: ## `destination-s3` Connector Test Results
0 tests   0 ✅  0s ⏱️
0 suites  0 💤
0 files    0 ❌

Results for commit ec4760e2.

[test-results]:data:application/gzip;base64,H4sIAHUqHWgC/1WMQQ7CIBBFr9KwdoEFS/EyBqZDMrEthoFV491taqOw++/95G0i0Iws7p28dIIL5R9MJblMcT1xPzLX+8EFoBVPejUiOJobgSnFdJpUVq5mnfvyv3ZwFTu4bkFcFso7CARtBol98La3eLNaDg5AKe+uqI03BsagplGJ9wfSF1T9+wAAAA==

:recycle: This comment has been updated with latest results.
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-05-07T23:46:43Z
body: ## `source-mysql` Connector Test Results
  8 files    8 suites   28s ⏱️
 46 tests  46 ✅ 0 💤 0 ❌
170 runs  170 ✅ 0 💤 0 ❌

Results for commit 15736f25.

[test-results]:data:application/gzip;base64,H4sIAOglHWgC/13MyQ7CIBSF4VdpWLuQqQVfxtwyJMS2GIZV47sLjVR0+f0nOTuybjER3QZxGVDMLp3QOUByfisk1WVJdWNjwz1mpf7Kwz1LuZ7Bglt+ggnBh08JeauPeGpoj335Ph7uDg/3f8qvq0sFCPOJjpZwAwQ00cBgZlzOUjHMtRSGckKBKoFeb0kaa4UCAQAA

:recycle: This comment has been updated with latest results.
author_association: CONTRIBUTOR
user: login: davinchia
created_at: 2025-05-08T01:33:25Z
body: Note: Gradle check will fail until we merge https://github.com/airbytehq/airbyte/pull/59722 in.
author_association: CONTRIBUTOR