Repository: debezium/debezium
Pull request #6254: DBZ-8702 Intorduce Debezium AI module with embeddings SMT
Author: login: vjuranek
Labels: name: stream/3.2
Created: 2025-03-26T12:30:30Z | Merged: 2025-04-02T10:10:09Z

https://issues.redhat.com/browse/DBZ-8702


Diff:
diff --git a/debezium-ai/debezium-ai-embeddings-minilm/pom.xml b/debezium-ai/debezium-ai-embeddings-minilm/pom.xml
new file mode 100644
index 00000000000..6b4c5211482
--- /dev/null
+++ b/debezium-ai/debezium-ai-embeddings-minilm/pom.xml
@@ -0,0 +1,112 @@
+<?xml version="1.0"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <parent>
+        <groupId>io.debezium</groupId>
+        <artifactId>debezium-ai</artifactId>
+        <version>3.1.0-SNAPSHOT</version>
+        <relativePath>../pom.xml</relativePath>
+    </parent>
+    <modelVersion>4.0.0</modelVersion>
+    <artifactId>debezium-ai-embeddings-minilm</artifactId>
+    <name>Debezium Embeddings MiniLM SMT</name>
+    <packaging>jar</packaging>
+
+    <dependencies>
+        <dependency>
+            <groupId>dev.langchain4j</groupId>
+            <artifactId>langchain4j-embeddings-all-minilm-l6-v2</artifactId>
+        </dependency>
+        <dependency>
+            <groupId>io.debezium</groupId>
+            <artifactId>debezium-ai-embeddings</artifactId>
+        </dependency>
+        <dependency>
+            <groupId>io.debezium</groupId>
+            <artifactId>debezium-core</artifactId>
+            <scope>provided</scope>
+        </dependency>
+
+        <!-- Test dependencies -->
+        <dependency>
+            <groupId>org.apache.kafka</groupId>
+            <artifactId>connect-transforms</artifactId>
+            <scope>provided</scope>
+        </dependency>
+        <dependency>
+            <groupId>io.debezium</groupId>
+            <artifactId>debezium-ai-embeddings</artifactId>
+            <type>test-jar</type>
+            <scope>test</scope>
+        </dependency>
+        <dependency>
+            <groupId>ch.qos.logback</groupId>
+            <artifactId>logback-classic</artifactId>
+            <scope>test</scope>
+        </dependency>
+        <dependency>
+            <groupId>junit</groupId>
+            <artifactId>junit</artifactId>
+            <scope>test</scope>
+        </dependency>
+        <dependency>
+            <groupId>org.assertj</groupId>
+            <artifactId>assertj-core</artifactId>
+            <scope>test</scope>
+        </dependency>
+    </dependencies>
+
+    <profiles>
+        <profile>
+            <id>assembly</id>
+            <activation>
+                <activeByDefault>false</activeByDefault>
+            </activation>
+            <build>
+                <plugins>
+                    <plugin>
+                        <groupId>org.apache.maven.plugins</groupId>
+                        <artifactId>maven-assembly-plugin</artifactId>
+                        <version>${version.assembly.plugin}</version>
+                        <dependencies>
+                            <dependency>
+                                <groupId>io.debezium</groupId>
+                                <artifactId>debezium-assembly-descriptors</artifactId>
+                                <version>${project.version}</version>
+                            </dependency>
+                        </dependencies>
+                        <executions>
+                            <execution>
+                                <id>default</id>
+                                <phase>package</phase>
+                                <goals>
+                                    <goal>single</goal>
+                                </goals>
+                                <configuration>
+                                    <finalName>${project.artifactId}-${project.version}</finalName>
+                                    <attach>true</attach>  <!-- we want attach & deploy these to Maven -->
+                                    <descriptorRefs>
+                                        <descriptorRef>${assembly.descriptor}</descriptorRef>
+                                    </descriptorRefs>
+                                    <tarLongFileMode>posix</tarLongFileMode>
+                                </configuration>
+                            </execution>
+                        </executions>
+                    </plugin>
+                </plugins>
+            </build>
+        </profile>
+        <profile>
+            <id>quick</id>
+            <activation>
+                <activeByDefault>false</activeByDefault>
+                <property>
+                    <name>quick</name>
+                </property>
+            </activation>
+            <properties>
+                <skipITs>true</skipITs>
+            </properties>
+        </profile>
+    </profiles>
+</project>
diff --git a/debezium-ai/debezium-ai-embeddings-minilm/src/main/java/io/debezium/ai/embeddings/MiniLmL6V2ModeFactory.java b/debezium-ai/debezium-ai-embeddings-minilm/src/main/java/io/debezium/ai/embeddings/MiniLmL6V2ModeFactory.java
new file mode 100644
index 00000000000..c4ff1349114
--- /dev/null
+++ b/debezium-ai/debezium-ai-embeddings-minilm/src/main/java/io/debezium/ai/embeddings/MiniLmL6V2ModeFactory.java
@@ -0,0 +1,42 @@
+/*
+ * Copyright Debezium Authors.
+ *
+ * Licensed under the Apache Software License version 2.0, available at http://www.apache.org/licenses/LICENSE-2.0
+ */
+package io.debezium.ai.embeddings;
+
+import org.apache.kafka.connect.connector.ConnectRecord;
+
+import io.debezium.config.Configuration;
+import io.debezium.config.Field;
+
+import dev.langchain4j.model.embedding.EmbeddingModel;
+import dev.langchain4j.model.embedding.onnx.allminilml6v2.AllMiniLmL6V2EmbeddingModel;
+
+/**
+ * Embedding model factory which uses <a href="https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2">MiniLmL6V2 model</a> for creating embeddings.
+ *
+ * @author vjuranek
+ */
+public class MiniLmL6V2ModeFactory<R extends ConnectRecord<R>> implements EmbeddingsModelFactory {
+
+    @Override
+    public Field.Set getConfigFields() {
+        return Field.setOf();
+    }
+
+    @Override
+    public void configure(Configuration config) {
+        // no-op
+    }
+
+    @Override
+    public void validateConfiguration() {
+        // no-op
+    }
+
+    @Override
+    public EmbeddingModel getModel() {
+        return new AllMiniLmL6V2EmbeddingModel();
+    }
+}
diff --git a/debezium-ai/debezium-ai-embeddings-minilm/src/main/resources/META-INF/services/io.debezium.ai.embeddings.EmbeddingsModelFactory b/debezium-ai/debezium-ai-embeddings-minilm/src/main/resources/META-INF/services/io.debezium.ai.embeddings.EmbeddingsModelFactory
new file mode 100644
index 00000000000..c979f1f7669
--- /dev/null
+++ b/debezium-ai/debezium-ai-embeddings-minilm/src/main/resources/META-INF/services/io.debezium.ai.embeddings.EmbeddingsModelFactory
@@ -0,0 +1 @@
+io.debezium.ai.embeddings.MiniLmL6V2ModeFactory
\ No newline at end of file
diff --git a/debezium-ai/debezium-ai-embeddings-minilm/src/test/java/io/debezium/ai/embeddings/EmbeddingsMiniLmL6V2Test.java b/debezium-ai/debezium-ai-embeddings-minilm/src/test/java/io/debezium/ai/embeddings/EmbeddingsMiniLmL6V2Test.java
new file mode 100644
index 00000000000..b60238c8807
--- /dev/null
+++ b/debezium-ai/debezium-ai-embeddings-minilm/src/test/java/io/debezium/ai/embeddings/EmbeddingsMiniLmL6V2Test.java
@@ -0,0 +1,38 @@
+/*
+ * Copyright Debezium Authors.
+ *
+ * Licensed under the Apache Software License version 2.0, available at http://www.apache.org/licenses/LICENSE-2.0
+ */
+package io.debezium.ai.embeddings;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.util.List;
+import java.util.Map;
+
+import org.apache.kafka.connect.data.Struct;
+import org.apache.kafka.connect.source.SourceRecord;
+import org.junit.Test;
+
+/**
+ * Basic tests for {@link FieldToEmbedding} SMT which uses {@link MiniLmL6V2ModeFactory}.
+ *
+ * @author vjuranek
+ */
+public class EmbeddingsMiniLmL6V2Test {
+    @Test
+    public void testMiniLmEmbeddings() {
+        FieldToEmbedding<SourceRecord> embeddingSmt = new FieldToEmbedding();
+        embeddingSmt.configure(Map.of(
+                "embeddings.field.source", "after.product",
+                "embeddings.field.embedding", "after.prod_embedding"));
+        SourceRecord transformedRecord = embeddingSmt.apply(FieldToEmbeddingTest.SOURCE_RECORD);
+
+        Struct payloadStruct = (Struct) transformedRecord.value();
+        assertThat(payloadStruct.getStruct("after").getString("product")).contains("a product");
+        List<Float> embeddings = payloadStruct.getStruct("after").getArray("prod_embedding");
+        assertThat(embeddings.size()).isEqualTo(384);
+        assertThat(embeddings).startsWith(-0.07157834f, 0.022438003f, -0.0236557f, -0.014294243f, 0.0048211957f, 0.020384153f, 0.20435654f, 0.057306267f, 0.05457874f,
+                -0.030588629f);
+    }
+}
diff --git a/debezium-ai/debezium-ai-embeddings-minilm/src/test/resources/logback-test.xml b/debezium-ai/debezium-ai-embeddings-minilm/src/test/resources/logback-test.xml
new file mode 100644
index 00000000000..e1a7291e8ce
--- /dev/null
+++ b/debezium-ai/debezium-ai-embeddings-minilm/src/test/resources/logback-test.xml
@@ -0,0 +1,23 @@
+<configuration>
+
+    <appender name="CONSOLE"
+        class="ch.qos.logback.core.ConsoleAppender">
+        <encoder>
+            <pattern>%d{ISO8601} %-5p  %X{dbz.connectorType}|%X{dbz.connectorName}|%X{dbz.connectorContext}  %m   [%c]%n</pattern>
+        </encoder>
+    </appender>
+
+    <root level="warn">
+        <appender-ref ref="CONSOLE" />
+    </root>
+
+    <!-- Set up the default logging to be INFO level, then override specific 
+        units -->
+    <logger name="io.debezium" level="trace" additivity="false">
+        <appender-ref ref="CONSOLE" />
+    </logger>
+
+    <logger name="io.debezium.ai.embeddings" level="trace" additivity="false">
+        <appender-ref ref="CONSOLE" />
+    </logger>
+</configuration>
diff --git a/debezium-ai/debezium-ai-embeddings-ollama/pom.xml b/debezium-ai/debezium-ai-embeddings-ollama/pom.xml
new file mode 100644
index 00000000000..763a9c4c50d
--- /dev/null
+++ b/debezium-ai/debezium-ai-embeddings-ollama/pom.xml
@@ -0,0 +1,141 @@
+<?xml version="1.0"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <parent>
+        <groupId>io.debezium</groupId>
+        <artifactId>debezium-ai</artifactId>
+        <version>3.1.0-SNAPSHOT</version>
+        <relativePath>../pom.xml</relativePath>
+    </parent>
+    <modelVersion>4.0.0</modelVersion>
+    <artifactId>debezium-ai-embeddings-ollama</artifactId>
+    <name>Debezium Embeddings Ollama SMT</name>
+    <packaging>jar</packaging>
+
+    <dependencies>
+        <dependency>
+            <groupId>dev.langchain4j</groupId>
+            <artifactId>langchain4j-ollama</artifactId>
+        </dependency>
+        <dependency>
+            <groupId>io.debezium</groupId>
+            <artifactId>debezium-ai-embeddings</artifactId>
+        </dependency>
+        <dependency>
+            <groupId>io.debezium</groupId>
+            <artifactId>debezium-core</artifactId>
+            <scope>provided</scope>
+        </dependency>
+
+        <!-- Test dependencies -->
+        <dependency>
+            <groupId>org.apache.kafka</groupId>
+            <artifactId>connect-transforms</artifactId>
+            <scope>provided</scope>
+        </dependency>
+        <dependency>
+            <groupId>io.debezium</groupId>
+            <artifactId>debezium-ai-embeddings</artifactId>
+            <type>test-jar</type>
+            <scope>test</scope>
+        </dependency>
+        <dependency>
+            <groupId>ch.qos.logback</groupId>
+            <artifactId>logback-classic</artifactId>
+            <scope>test</scope>
+        </dependency>
+        <dependency>
+            <groupId>junit</groupId>
+            <artifactId>junit</artifactId>
+            <scope>test</scope>
+        </dependency>
+        <dependency>
+            <groupId>org.assertj</groupId>
+            <artifactId>assertj-core</artifactId>
+            <scope>test</scope>
+        </dependency>
+        <dependency>
+            <groupId>org.testcontainers</groupId>
+            <artifactId>ollama</artifactId>
+            <scope>test</scope>
+        </dependency>
+    </dependencies>
+
+    <build>
+        <plugins>
+            <!-- Integration test -->
+            <plugin>
+                <groupId>org.apache.maven.plugins</groupId>
+                <artifactId>maven-failsafe-plugin</artifactId>
+                <executions>
+                    <execution>
+                        <id>integration-test</id>
+                        <goals>
+                            <goal>integration-test</goal>
+                        </goals>
+                    </execution>
+                    <execution>
+                        <id>verify</id>
+                        <goals>
+                            <goal>verify</goal>
+                        </goals>
+                    </execution>
+                </executions>
+            </plugin>
+        </plugins>
+    </build>
+
+    <profiles>
+        <profile>
+            <id>assembly</id>
+            <activation>
+                <activeByDefault>false</activeByDefault>
+            </activation>
+            <build>
+                <plugins>
+                    <plugin>
+                        <groupId>org.apache.maven.plugins</groupId>
+                        <artifactId>maven-assembly-plugin</artifactId>
+                        <version>${version.assembly.plugin}</version>
+                        <dependencies>
+                            <dependency>
+                                <groupId>io.debezium</groupId>
+                                <artifactId>debezium-assembly-descriptors</artifactId>
+                                <version>${project.version}</version>
+                            </dependency>
+                        </dependencies>
+                        <executions>
+                            <execution>
+                                <id>default</id>
+                                <phase>package</phase>
+                                <goals>
+                                    <goal>single</goal>
+                                </goals>
+                                <configuration>
+                                    <finalName>${project.artifactId}-${project.version}</finalName>
+                                    <attach>true</attach>  <!-- we want attach & deploy these to Maven -->
+                                    <descriptorRefs>
+                                        <descriptorRef>${assembly.descriptor}</descriptorRef>
+                                    </descriptorRefs>
+                                    <tarLongFileMode>posix</tarLongFileMode>
+                                </configuration>
+                            </execution>
+                        </executions>
+                    </plugin>
+                </plugins>
+            </build>
+        </profile>
+        <profile>
+            <id>quick</id>
+            <activation>
+                <activeByDefault>false</activeByDefault>
+                <property>
+                    <name>quick</name>
+                </property>
+            </activation>
+            <properties>
+                <skipITs>true</skipITs>
+            </properties>
+        </profile>
+    </profiles>
+</project>
diff --git a/debezium-ai/debezium-ai-embeddings-ollama/src/main/java/io/debezium/ai/embeddings/OllamaModelFactory.java b/debezium-ai/debezium-ai-embeddings-ollama/src/main/java/io/debezium/ai/embeddings/OllamaModelFactory.java
new file mode 100644
index 00000000000..c71e4a59c9a
--- /dev/null
+++ b/debezium-ai/debezium-ai-embeddings-ollama/src/main/java/io/debezium/ai/embeddings/OllamaModelFactory.java
@@ -0,0 +1,79 @@
+/*
+ * Copyright Debezium Authors.
+ *
+ * Licensed under the Apache Software License version 2.0, available at http://www.apache.org/licenses/LICENSE-2.0
+ */
+package io.debezium.ai.embeddings;
+
+import static io.debezium.ai.embeddings.FieldToEmbedding.EMBEDDINGS_PREFIX;
+import static java.lang.String.format;
+
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigException;
+import org.apache.kafka.connect.connector.ConnectRecord;
+
+import io.debezium.config.Configuration;
+import io.debezium.config.Field;
+
+import dev.langchain4j.model.embedding.EmbeddingModel;
+import dev.langchain4j.model.ollama.OllamaEmbeddingModel;
+
+/**
+ * {@link EmbeddingsModelFactory}  for {@link FieldToEmbedding} SMT, which uses <a href="https://ollama.com/">Ollama</a> for serving embedding models.
+ *
+ * @author vjuranek
+ */
+public class OllamaModelFactory<R extends ConnectRecord<R>> implements EmbeddingsModelFactory {
+
+    public static final String OLLAMA_PREFIX = EMBEDDINGS_PREFIX + "ollama.";
+
+    private static final Field OLLAMA_BASE_URL = Field.create(OLLAMA_PREFIX + "url")
+            .withDisplayName("Ollama server URL.")
+            .withType(ConfigDef.Type.STRING)
+            .withWidth(ConfigDef.Width.SHORT)
+            .withImportance(ConfigDef.Importance.HIGH)
+            .withDescription("Base URL of Ollama server.")
+            .required();
+
+    private static final Field MODEL_NAME = Field.create(OLLAMA_PREFIX + "model.name")
+            .withDisplayName("Model name.")
+            .withType(ConfigDef.Type.STRING)
+            .withWidth(ConfigDef.Width.SHORT)
+            .withImportance(ConfigDef.Importance.HIGH)
+            .withDescription("Name of the model which should be served by Ollama server.")
+            .required();
+
+    public static final Field.Set ALL_FIELDS = Field.setOf(OLLAMA_BASE_URL, MODEL_NAME);
+
+    private String baseUrl;
+    private String modelName;
+
+    @Override
+    public Field.Set getConfigFields() {
+        return ALL_FIELDS;
+    }
+
+    @Override
+    public void configure(Configuration config) {
+        baseUrl = config.getString(OLLAMA_BASE_URL);
+        modelName = config.getString(MODEL_NAME);
+    }
+
+    @Override
+    public void validateConfiguration() {
+        if (baseUrl == null || baseUrl.isBlank()) {
+            throw new ConfigException(format("'%s' must be set to non-empty value.", OLLAMA_BASE_URL));
+        }
+        if (modelName.isBlank() || modelName.isBlank()) {
+            throw new ConfigException(format("'%s' must be set to non-empty value.", MODEL_NAME));
+        }
+    }
+
+    @Override
+    public EmbeddingModel getModel() {
+        return OllamaEmbeddingModel.builder()
+                .baseUrl(baseUrl)
+                .modelName(modelName)
+                .build();
+    }
+}
diff --git a/debezium-ai/debezium-ai-embeddings-ollama/src/main/resources/META-INF/services/io.debezium.ai.embeddings.EmbeddingsModelFactory b/debezium-ai/debezium-ai-embeddings-ollama/src/main/resources/META-INF/services/io.debezium.ai.embeddings.EmbeddingsModelFactory
new file mode 100644
index 00000000000..d1b2373d469
--- /dev/null
+++ b/debezium-ai/debezium-ai-embeddings-ollama/src/main/resources/META-INF/services/io.debezium.ai.embeddings.EmbeddingsModelFactory
@@ -0,0 +1 @@
+io.debezium.ai.embeddings.OllamaModelFactory
\ No newline at end of file
diff --git a/debezium-ai/debezium-ai-embeddings-ollama/src/test/java/io/debezium/ai/embeddings/EmbeddingsOllamaIT.java b/debezium-ai/debezium-ai-embeddings-ollama/src/test/java/io/debezium/ai/embeddings/EmbeddingsOllamaIT.java
new file mode 100644
index 00000000000..9173cd9c2ab
--- /dev/null
+++ b/debezium-ai/debezium-ai-embeddings-ollama/src/test/java/io/debezium/ai/embeddings/EmbeddingsOllamaIT.java
@@ -0,0 +1,66 @@
+/*
+ * Copyright Debezium Authors.
+ *
+ * Licensed under the Apache Software License version 2.0, available at http://www.apache.org/licenses/LICENSE-2.0
+ */
+package io.debezium.ai.embeddings;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.kafka.connect.data.Struct;
+import org.apache.kafka.connect.source.SourceRecord;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.testcontainers.ollama.OllamaContainer;
+import org.testcontainers.utility.DockerImageName;
+
+/**
+ * Integrations tests for {@link FieldToEmbedding} SMT which uses {@link OllamaModelFactory}.
+ *
+ * @author vjuranek
+ */
+public class EmbeddingsOllamaIT {
+    private static final String OLLAMA_IMAGE_NAME = "mirror.gcr.io/ollama/ollama:0.6.2";
+    private static final String OLLAMA_TEST_MODEL = "all-minilm";
+
+    private static final OllamaContainer ollama = new OllamaContainer(
+            DockerImageName.parse(OLLAMA_IMAGE_NAME).asCompatibleSubstituteFor("ollama/ollama"))
+            .withStartupTimeout(Duration.ofSeconds(180));
+
+    private final FieldToEmbedding<SourceRecord> embeddingSmt = new FieldToEmbedding();
+
+    @BeforeClass
+    public static void startDatabase() {
+        ollama.start();
+    }
+
+    @AfterClass
+    public static void stopDatabase() {
+        ollama.stop();
+    }
+
+    @Test
+    public void testOllamaEmbeddings() throws InterruptedException, IOException {
+        ollama.execInContainer("ollama", "pull", OLLAMA_TEST_MODEL);
+
+        embeddingSmt.configure(Map.of(
+                "embeddings.field.source", "after.product",
+                "embeddings.field.embedding", "after.prod_embedding",
+                "embeddings.ollama.url", ollama.getEndpoint(),
+                "embeddings.ollama.model.name", OLLAMA_TEST_MODEL));
+        SourceRecord transformedRecord = embeddingSmt.apply(FieldToEmbeddingTest.SOURCE_RECORD);
+
+        Struct payloadStruct = (Struct) transformedRecord.value();
+        assertThat(payloadStruct.getStruct("after").getString("product")).contains("a product");
+        List<Float> embeddings = payloadStruct.getStruct("after").getArray("prod_embedding");
+        assertThat(embeddings.size()).isEqualTo(384);
+        assertThat(embeddings).startsWith(-0.07157089f, 0.022460647f, -0.02369636f, -0.0143798785f, 0.0048304256f, 0.020285256f, 0.20442571f, 0.057290666f, 0.054607023f,
+                -0.030602805f);
+    }
+}
diff --git a/debezium-ai/debezium-ai-embeddings-ollama/src/test/resources/logback-test.xml b/debezium-ai/debezium-ai-embeddings-ollama/src/test/resources/logback-test.xml
new file mode 100644
index 00000000000..e1a7291e8ce
--- /dev/null
+++ b/debezium-ai/debezium-ai-embeddings-ollama/src/test/resources/logback-test.xml
@@ -0,0 +1,23 @@
+<configuration>
+
+    <appender name="CONSOLE"
+        class="ch.qos.logback.core.ConsoleAppender">
+        <encoder>
+            <pattern>%d{ISO8601} %-5p  %X{dbz.connectorType}|%X{dbz.connectorName}|%X{dbz.connectorContext}  %m   [%c]%n</pattern>
+        </encoder>
+    </appender>
+
+    <root level="warn">
+        <appender-ref ref="CONSOLE" />
+    </root>
+
+    <!-- Set up the default logging to be INFO level, then override specific 
+        units -->
+    <logger name="io.debezium" level="trace" additivity="false">
+        <appender-ref ref="CONSOLE" />
+    </logger>
+
+    <logger name="io.debezium.ai.embeddings" level="trace" additivity="false">
+        <appender-ref ref="CONSOLE" />
+    </logger>
+</configuration>
diff --git a/debezium-ai/debezium-ai-embeddings/pom.xml b/debezium-ai/debezium-ai-embeddings/pom.xml
new file mode 100644
index 00000000000..dd5f1b663b0
--- /dev/null
+++ b/debezium-ai/debezium-ai-embeddings/pom.xml
@@ -0,0 +1,68 @@
+<?xml version="1.0"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <parent>
+        <groupId>io.debezium</groupId>
+        <artifactId>debezium-ai</artifactId>
+        <version>3.1.0-SNAPSHOT</version>
+        <relativePath>../pom.xml</relativePath>
+    </parent>
+    <modelVersion>4.0.0</modelVersion>
+    <artifactId>debezium-ai-embeddings</artifactId>
+    <name>Debezium Embeddings SMT</name>
+    <packaging>jar</packaging>
+
+    <dependencies>
+        <dependency>
+            <groupId>dev.langchain4j</groupId>
+            <artifactId>langchain4j-core</artifactId>
+        </dependency>
+        <dependency>
+            <groupId>io.debezium</groupId>
+            <artifactId>debezium-core</artifactId>
+            <scope>provided</scope>
+        </dependency>
+        <dependency>
+            <groupId>org.apache.kafka</groupId>
+            <artifactId>connect-api</artifactId>
+            <scope>provided</scope>
+        </dependency>
+        <dependency>
+            <groupId>org.apache.kafka</groupId>
+            <artifactId>connect-transforms</artifactId>
+            <scope>provided</scope>
+        </dependency>
+
+        <!-- Test dependencies -->
+        <dependency>
+            <groupId>ch.qos.logback</groupId>
+            <artifactId>logback-classic</artifactId>
+            <scope>test</scope>
+        </dependency>
+        <dependency>
+            <groupId>junit</groupId>
+            <artifactId>junit</artifactId>
+            <scope>test</scope>
+        </dependency>
+        <dependency>
+            <groupId>org.assertj</groupId>
+            <artifactId>assertj-core</artifactId>
+            <scope>test</scope>
+        </dependency>
+    </dependencies>
+
+    <profiles>
+        <profile>
+            <id>quick</id>
+            <activation>
+                <activeByDefault>false</activeByDefault>
+                <property>
+                    <name>quick</name>
+                </property>
+            </activation>
+            <properties>
+                <skipITs>true</skipITs>
+            </properties>
+        </profile>
+    </profiles>
+</project>
diff --git a/debezium-ai/debezium-ai-embeddings/src/main/java/io/debezium/ai/embeddings/EmbeddingsModelFactory.java b/debezium-ai/debezium-ai-embeddings/src/main/java/io/debezium/ai/embeddings/EmbeddingsModelFactory.java
new file mode 100644
index 00000000000..cf78e70aa46
--- /dev/null
+++ b/debezium-ai/debezium-ai-embeddings/src/main/java/io/debezium/ai/embeddings/EmbeddingsModelFactory.java
@@ -0,0 +1,30 @@
+/*
+ * Copyright Debezium Authors.
+ *
+ * Licensed under the Apache Software License version 2.0, available at http://www.apache.org/licenses/LICENSE-2.0
+ */
+
+package io.debezium.ai.embeddings;
+
+import io.debezium.config.Configuration;
+import io.debezium.config.Field;
+
+import dev.langchain4j.model.embedding.EmbeddingModel;
+
+/**
+ * Implementations of this interface provide various kind of embedding models as provided by various modules of LangChain4j.
+ * Appropriate factory is loaded via SPI in {@link FieldToEmbedding} SMT.
+ * Model configuration is part of SMT configuration.
+ *
+ * @author vjuranek
+ */
+public interface EmbeddingsModelFactory {
+
+    Field.Set getConfigFields();
+
+    void configure(Configuration config);
+
+    void validateConfiguration();
+
+    EmbeddingModel getModel();
+}
diff --git a/debezium-ai/debezium-ai-embeddings/src/main/java/io/debezium/ai/embeddings/FieldToEmbedding.java b/debezium-ai/debezium-ai-embeddings/src/main/java/io/debezium/ai/embeddings/FieldToEmbedding.java
new file mode 100644
index 00000000000..a8e532989c7
--- /dev/null
+++ b/debezium-ai/debezium-ai-embeddings/src/main/java/io/debezium/ai/embeddings/FieldToEmbedding.java
@@ -0,0 +1,211 @@
+/*
+ * Copyright Debezium Authors.
+ *
+ * Licensed under the Apache Software License version 2.0, available at http://www.apache.org/licenses/LICENSE-2.0
+ */
+
+package io.debezium.ai.embeddings;
+
+import static java.lang.String.format;
+import static org.apache.kafka.connect.transforms.util.Requirements.requireStruct;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.ServiceLoader;
+
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigException;
+import org.apache.kafka.connect.components.Versioned;
+import org.apache.kafka.connect.connector.ConnectRecord;
+import org.apache.kafka.connect.data.Schema;
+import org.apache.kafka.connect.data.Struct;
+import org.apache.kafka.connect.transforms.Transformation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import io.debezium.DebeziumException;
+import io.debezium.Module;
+import io.debezium.config.Configuration;
+import io.debezium.config.Field;
+import io.debezium.data.vector.FloatVector;
+import io.debezium.transforms.ConnectRecordUtil;
+import io.debezium.transforms.SmtManager;
+import io.debezium.util.BoundedConcurrentHashMap;
+
+import dev.langchain4j.data.embedding.Embedding;
+import dev.langchain4j.data.segment.TextSegment;
+import dev.langchain4j.model.embedding.EmbeddingModel;
+
+/**
+ * Single message transform which appends to the record payload embedding of selected {@link String} field.
+ * Embedding model is provided via appropriate {@link EmbeddingsModelFactory} factory, which under the hood
+ * uses <a href="https://docs.langchain4j.dev/">LangChain4j</a> project. Model factory class is loaded via
+ * SPI. You have to place one of the Debezium AI Embeddings modules with model of your choice on the class
+ * path or implement your own model factory provider and place it on the class path.
+ *
+ * @author vjuranek
+ */
+public class FieldToEmbedding<R extends ConnectRecord<R>> implements Transformation<R>, Versioned {
+
+    private static final Logger LOGGER = LoggerFactory.getLogger(FieldToEmbedding.class);
+
+    public static final String EMBEDDINGS_PREFIX = "embeddings.";
+
+    private static final Field TEXT_FIELD = Field.create(EMBEDDINGS_PREFIX + "field.source")
+            .withDisplayName("Name of the record field from which embeddings should be created.")
+            .withType(ConfigDef.Type.STRING)
+            .withWidth(ConfigDef.Width.SHORT)
+            .withImportance(ConfigDef.Importance.HIGH)
+            .required()
+            .withDescription("Name of the field of the record which content will be used as an input for embeddings. Supports also nested fields.");
+
+    private static final Field EMBEDDGINS_FIELD = Field.create(EMBEDDINGS_PREFIX + "field.embedding")
+            .withDisplayName("Name of the field which would contain the embeddings of the input field.")
+            .withType(ConfigDef.Type.STRING)
+            .withWidth(ConfigDef.Width.SHORT)
+            .withImportance(ConfigDef.Importance.HIGH)
+            .withDescription(
+                    "Name of the field which which will be appended to the record and which would contain the embeddings of the content `filed.source` field. Supports also nested fields.");
+
+    private static final Schema EMBEDDING_SCHEMA = FloatVector.schema();
+    private static final EmbeddingsModelFactory MODEL_FACTORY = EmbeddingsModelFactoryLoader.getModelFactory();
+    public static final Field.Set ALL_FIELDS = Field.setOf(TEXT_FIELD, EMBEDDGINS_FIELD).with(MODEL_FACTORY.getConfigFields());
+
+    private SmtManager<R> smtManager;
+    private String sourceField;
+    private String embeddingsField;
+    private List<String> sourceFiledPath;
+    private EmbeddingModel model;
+
+    private static final String NESTING_SPLIT_REG_EXP = "\\.";
+    private static final int CACHE_SIZE = 64;
+    private final BoundedConcurrentHashMap<Schema, Schema> schemaUpdateCache = new BoundedConcurrentHashMap<>(CACHE_SIZE);
+
+    @Override
+    public void configure(Map<String, ?> configs) {
+        final Configuration config = Configuration.from(configs);
+        smtManager = new SmtManager<>(config);
+        smtManager.validate(config, ALL_FIELDS);
+
+        sourceField = config.getString(TEXT_FIELD);
+        embeddingsField = config.getString(EMBEDDGINS_FIELD);
+        MODEL_FACTORY.configure(config);
+        validateConfiguration();
+
+        sourceFiledPath = Arrays.asList(sourceField.split(NESTING_SPLIT_REG_EXP));
+        model = MODEL_FACTORY.getModel();
+    }
+
+    @Override
+    public R apply(R record) {
+        if (record.value() == null || !smtManager.isValidEnvelope(record)) {
+            LOGGER.trace("Record {} has null value of invalid envelope and will be skipped.", record.value());
+            return record;
+        }
+
+        final String text = getSourceString(record);
+        return text == null ? record : buildUpdatedRecord(record, text);
+    }
+
+    @Override
+    public ConfigDef config() {
+        final ConfigDef config = new ConfigDef();
+        Field.group(config, null, TEXT_FIELD, EMBEDDGINS_FIELD);
+        return config;
+    }
+
+    @Override
+    public void close() {
+    }
+
+    @Override
+    public String version() {
+        return Module.version();
+    }
+
+    protected void validateConfiguration() {
+        if (sourceField == null || sourceField.isBlank()) {
+            throw new ConfigException(format("'%s' must be set to non-empty value.", TEXT_FIELD));
+        }
+        MODEL_FACTORY.validateConfiguration();
+    }
+
+    /**
+     *
+     * Based on the configuration, obtains value of the record field from which the embeddings will be computed.
+     * This field has to be of type {@link String}.
+     */
+    protected String getSourceString(R record) {
+        if (record.value() != null && smtManager.isValidEnvelope(record) && record.valueSchema().type() == Schema.Type.STRUCT) {
+            Struct struct = requireStruct(record.value(), "Obtaining source field for embeddings");
+            for (int i = 0; i < sourceFiledPath.size() - 1; i++) {
+                if (struct.schema().type() == Schema.Type.STRUCT) {
+                    struct = struct.getStruct(sourceFiledPath.get(i));
+                }
+                else {
+                    throw new IllegalArgumentException(format("Invalid field name %s, %s is not struct.", sourceField, struct.schema().name()));
+                }
+            }
+            return struct.getString(sourceFiledPath.getLast());
+        }
+        else {
+            LOGGER.debug("Skipping record {}, it has either null value or invalid structure", record);
+        }
+        return null;
+    }
+
+    /**
+     * Copies the original record and appends to it embeddings of the text contained in the source field of the records.
+     */
+    protected R buildUpdatedRecord(R original, String text) {
+        final Struct value = requireStruct(original.value(), "Original value must be struct");
+        final TextSegment segment = TextSegment.from(text);
+        final Embedding embedding = model.embed(segment).content();
+
+        final Schema updatedSchema;
+        final Object updatedValue;
+        if (embeddingsField == null) {
+            updatedSchema = EMBEDDING_SCHEMA;
+            updatedValue = embedding.vectorAsList();
+        }
+        else {
+            final List<ConnectRecordUtil.NewEntry> newEntries = List.of(new ConnectRecordUtil.NewEntry(embeddingsField, EMBEDDING_SCHEMA, embedding.vectorAsList()));
+            updatedSchema = schemaUpdateCache.computeIfAbsent(value.schema(), valueSchema -> ConnectRecordUtil.makeNewSchema(valueSchema, newEntries));
+            updatedValue = ConnectRecordUtil.makeUpdatedValue(value, newEntries, updatedSchema);
+        }
+
+        return original.newRecord(
+                original.topic(),
+                original.kafkaPartition(),
+                original.keySchema(),
+                original.key(),
+                updatedSchema,
+                updatedValue,
+                original.timestamp(),
+                original.headers());
+    }
+
+    /**
+     * {@link EmbeddingsModelFactory} loader which loads model factory supplied by the user via SPI.
+     */
+    public static class EmbeddingsModelFactoryLoader<R extends ConnectRecord<R>> {
+        private static final Logger LOGGER = LoggerFactory.getLogger(FieldToEmbedding.EmbeddingsModelFactoryLoader.class);
+
+        static EmbeddingsModelFactory getModelFactory() {
+            final ServiceLoader<EmbeddingsModelFactory> loader = ServiceLoader.load(EmbeddingsModelFactory.class);
+            final Optional<EmbeddingsModelFactory> factory = loader.findFirst();
+            if (factory.isEmpty()) {
+                throw new DebeziumException("No implementation of Debezium embeddings model factory found.");
+            }
+            if (loader.stream().count() > 1) {
+                LOGGER.warn(
+                        "More then one Debezium embeddings model factory found. Order of loading is not defined and you may load different factory than you intended.");
+                LOGGER.warn("Found following factories:");
+                loader.stream().forEach(f -> LOGGER.warn(f.get().getClass().getName()));
+            }
+            return factory.get();
+        }
+    }
+}
diff --git a/debezium-ai/debezium-ai-embeddings/src/test/java/io/debezium/ai/embeddings/DummyModelFactory.java b/debezium-ai/debezium-ai-embeddings/src/test/java/io/debezium/ai/embeddings/DummyModelFactory.java
new file mode 100644
index 00000000000..ee63209e4ed
--- /dev/null
+++ b/debezium-ai/debezium-ai-embeddings/src/test/java/io/debezium/ai/embeddings/DummyModelFactory.java
@@ -0,0 +1,56 @@
+/*
+ * Copyright Debezium Authors.
+ *
+ * Licensed under the Apache Software License version 2.0, available at http://www.apache.org/licenses/LICENSE-2.0
+ */
+
+package io.debezium.ai.embeddings;
+
+import java.util.List;
+
+import io.debezium.config.Configuration;
+import io.debezium.config.Field;
+
+import dev.langchain4j.data.embedding.Embedding;
+import dev.langchain4j.data.segment.TextSegment;
+import dev.langchain4j.model.embedding.EmbeddingModel;
+import dev.langchain4j.model.output.Response;
+
+/**
+ * Dummy model factory for testing.
+ *
+ * @author vjuranek
+ */
+public class DummyModelFactory implements EmbeddingsModelFactory {
+
+    @Override
+    public Field.Set getConfigFields() {
+        return Field.setOf();
+    }
+
+    @Override
+    public void configure(Configuration config) {
+        // no-op
+    }
+
+    @Override
+    public void validateConfiguration() {
+        // no-op
+    }
+
+    @Override
+    public EmbeddingModel getModel() {
+        return new DummyEmbeddingModel();
+    }
+
+    /**
+     * Implementation of {@link EmbeddingModel} which returns constant vector for any input.
+     */
+    private static class DummyEmbeddingModel implements EmbeddingModel {
+        @Override
+        public Response<List<Embedding>> embedAll(List<TextSegment> textSegments) {
+            float[] dummyVector = new float[]{ 0.f, 1.f, 2.f, 3.f };
+            return new Response<>(List.of(new Embedding(dummyVector)));
+        }
+    }
+}
diff --git a/debezium-ai/debezium-ai-embeddings/src/test/java/io/debezium/ai/embeddings/FieldToEmbeddingTest.java b/debezium-ai/debezium-ai-embeddings/src/test/java/io/debezium/ai/embeddings/FieldToEmbeddingTest.java
new file mode 100644
index 00000000000..88ad7b97bcb
--- /dev/null
+++ b/debezium-ai/debezium-ai-embeddings/src/test/java/io/debezium/ai/embeddings/FieldToEmbeddingTest.java
@@ -0,0 +1,86 @@
+/*
+ * Copyright Debezium Authors.
+ *
+ * Licensed under the Apache Software License version 2.0, available at http://www.apache.org/licenses/LICENSE-2.0
+ */
+
+package io.debezium.ai.embeddings;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+import java.time.Instant;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.kafka.connect.data.Schema;
+import org.apache.kafka.connect.data.SchemaBuilder;
+import org.apache.kafka.connect.data.Struct;
+import org.apache.kafka.connect.source.SourceRecord;
+import org.junit.Test;
+
+import io.debezium.data.Envelope;
+
+/**
+ * Basic tests of {@link FieldToEmbedding} SMT.
+ *
+ * @author vjuranek
+ */
+public class FieldToEmbeddingTest {
+    public static final Schema VALUE_SCHEMA = SchemaBuilder.struct()
+            .name("mysql.inventory.products.Value")
+            .field("id", Schema.INT64_SCHEMA)
+            .field("price", Schema.FLOAT32_SCHEMA)
+            .field("product", Schema.STRING_SCHEMA)
+            .build();
+
+    public static final Struct ROW = new Struct(VALUE_SCHEMA)
+            .put("id", 101L)
+            .put("price", 20.0F)
+            .put("product", "a product");
+
+    public static final Envelope ENVELOPE = Envelope.defineSchema()
+            .withName("mysql.inventory.products.Envelope")
+            .withRecord(VALUE_SCHEMA)
+            .withSource(Schema.STRING_SCHEMA)
+            .build();
+
+    public static final Struct PAYLOAD = ENVELOPE.create(ROW, null, Instant.now());
+    public static final SourceRecord SOURCE_RECORD = new SourceRecord(new HashMap<>(), new HashMap<>(), "topic", ENVELOPE.schema(), PAYLOAD);
+
+    @Test
+    public void testNonNestedFieldIsEmbeddedNonNested() {
+        FieldToEmbedding<SourceRecord> embeddingSmt = new FieldToEmbedding();
+        embeddingSmt.configure(Map.of(
+                "embeddings.field.source", "op",
+                "embeddings.field.embedding", "op_embedding"));
+        SourceRecord transformedRecord = embeddingSmt.apply(SOURCE_RECORD);
+
+        Struct payloadStruct = (Struct) transformedRecord.value();
+        assertThat(payloadStruct.getString("op")).isEqualTo("c");
+        assertThat(payloadStruct.getArray("op_embedding")).contains(0.0f, 1.0f, 2.0f, 3.0f);
+    }
+
+    @Test
+    public void testNestedFieldIsEmbeddedNested() {
+        FieldToEmbedding<SourceRecord> embeddingSmt = new FieldToEmbedding();
+        embeddingSmt.configure(Map.of(
+                "embeddings.field.source", "after.product",
+                "embeddings.field.embedding", "after.prod_embedding"));
+        SourceRecord transformedRecord = embeddingSmt.apply(SOURCE_RECORD);
+
+        Struct payloadStruct = (Struct) transformedRecord.value();
+        assertThat(payloadStruct.getStruct("after").getString("product")).contains("a product");
+        assertThat(payloadStruct.getStruct("after").getArray("prod_embedding")).contains(0.0f, 1.0f, 2.0f, 3.0f);
+    }
+
+    @Test
+    public void testNoEmbeddingsConfProvided() {
+        FieldToEmbedding<SourceRecord> embeddingSmt = new FieldToEmbedding();
+        embeddingSmt.configure(Map.of("embeddings.field.source", "after.product"));
+        SourceRecord transformedRecord = embeddingSmt.apply(SOURCE_RECORD);
+
+        List<Float> payloadStruct = (List<Float>) transformedRecord.value();
+        assertThat(payloadStruct).contains(0.0f, 1.0f, 2.0f, 3.0f);
+    }
+}
diff --git a/debezium-ai/debezium-ai-embeddings/src/test/resources/META-INF/services/io.debezium.ai.embeddings.EmbeddingsModelFactory b/debezium-ai/debezium-ai-embeddings/src/test/resources/META-INF/services/io.debezium.ai.embeddings.EmbeddingsModelFactory
new file mode 100644
index 00000000000..5d80cfccfb2
--- /dev/null
+++ b/debezium-ai/debezium-ai-embeddings/src/test/resources/META-INF/services/io.debezium.ai.embeddings.EmbeddingsModelFactory
@@ -0,0 +1 @@
+io.debezium.ai.embeddings.DummyModelFactory
\ No newline at end of file
diff --git a/debezium-ai/debezium-ai-embeddings/src/test/resources/logback-test.xml b/debezium-ai/debezium-ai-embeddings/src/test/resources/logback-test.xml
new file mode 100644
index 00000000000..e1a7291e8ce
--- /dev/null
+++ b/debezium-ai/debezium-ai-embeddings/src/test/resources/logback-test.xml
@@ -0,0 +1,23 @@
+<configuration>
+
+    <appender name="CONSOLE"
+        class="ch.qos.logback.core.ConsoleAppender">
+        <encoder>
+            <pattern>%d{ISO8601} %-5p  %X{dbz.connectorType}|%X{dbz.connectorName}|%X{dbz.connectorContext}  %m   [%c]%n</pattern>
+        </encoder>
+    </appender>
+
+    <root level="warn">
+        <appender-ref ref="CONSOLE" />
+    </root>
+
+    <!-- Set up the default logging to be INFO level, then override specific 
+        units -->
+    <logger name="io.debezium" level="trace" additivity="false">
+        <appender-ref ref="CONSOLE" />
+    </logger>
+
+    <logger name="io.debezium.ai.embeddings" level="trace" additivity="false">
+        <appender-ref ref="CONSOLE" />
+    </logger>
+</configuration>
diff --git a/debezium-ai/pom.xml b/debezium-ai/pom.xml
new file mode 100644
index 00000000000..e7e1edd5870
--- /dev/null
+++ b/debezium-ai/pom.xml
@@ -0,0 +1,33 @@
+<?xml version="1.0"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <parent>
+        <groupId>io.debezium</groupId>
+        <artifactId>debezium-parent</artifactId>
+        <version>3.1.0-SNAPSHOT</version>
+        <relativePath>../debezium-parent/pom.xml</relativePath>
+    </parent>
+    <modelVersion>4.0.0</modelVersion>
+    <artifactId>debezium-ai</artifactId>
+    <version>3.1.0-SNAPSHOT</version>
+    <name>Debezium AI</name>
+    <packaging>pom</packaging>
+
+    <properties>
+        <!-- JDK version is controlled by Debezium Parent, do not change! -->
+        <maven.compiler.source>${debezium.java.source}</maven.compiler.source>
+        <maven.compiler.target>${debezium.java.specific.target}</maven.compiler.target>
+        <maven.compiler.release>${debezium.java.specific.target}</maven.compiler.release>
+        <maven.compiler.testSource>${debezium.java.source}</maven.compiler.testSource>
+        <maven.compiler.testTarget>${debezium.java.specific.target}</maven.compiler.testTarget>
+        <maven.compiler.testRelease>${debezium.java.specific.target}</maven.compiler.testRelease>
+        <!-- Assembly configuration -->
+        <assembly.descriptor>ai-embeddings-distribution</assembly.descriptor>
+    </properties>
+
+    <modules>
+        <module>debezium-ai-embeddings</module>
+        <module>debezium-ai-embeddings-minilm</module>
+        <module>debezium-ai-embeddings-ollama</module>
+    </modules>
+</project>
diff --git a/debezium-assembly-descriptors/src/main/resources/assemblies/ai-embeddings-distribution.xml b/debezium-assembly-descriptors/src/main/resources/assemblies/ai-embeddings-distribution.xml
new file mode 100644
index 00000000000..42b93535747
--- /dev/null
+++ b/debezium-assembly-descriptors/src/main/resources/assemblies/ai-embeddings-distribution.xml
@@ -0,0 +1,63 @@
+<?xml version="1.0"?>
+<assembly xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.2"
+          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+          xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.2 http://maven.apache.org/xsd/assembly-1.1.2.xsd">
+    <id>embeddings</id>
+    <formats>
+        <format>tar.gz</format>
+        <format>zip</format>
+    </formats>
+    <includeBaseDirectory>false</includeBaseDirectory>
+    <dependencySets>
+        <dependencySet>
+            <outputDirectory>${project.artifactId}</outputDirectory>
+            <unpack>false</unpack>
+            <scope>runtime</scope>
+            <useProjectArtifact>false</useProjectArtifact>
+            <useTransitiveFiltering>true</useTransitiveFiltering>
+            <excludes>
+                <!-- Exclude dependencies of Kafka APIs, since they will be available in the runtime -->
+                <exclude>com.fasterxml.jackson.core:jackson-core:*</exclude>
+                <exclude>com.fasterxml.jackson.core:jackson-databind:*</exclude>
+                <exclude>com.fasterxml.jackson.core:jackson-annotations:*</exclude>
+                <exclude>com.fasterxml.jackson.datatype:jackson-datatype-jsr310:*</exclude>
+
+                <!-- Exclude guava dependencies -->
+                <exclude>com.google.guava:listenablefuture:*</exclude>
+
+                <!-- Exclude dependencies with incorrect scope -->
+                <exclude>org.checkerframework:checker-qual:*</exclude>
+            </excludes>
+        </dependencySet>
+        <dependencySet>
+            <outputDirectory>${project.artifactId}</outputDirectory>
+            <unpack>false</unpack>
+            <includes>
+                <include>${project.groupId}:${project.artifactId}:*</include>
+            </includes>
+        </dependencySet>
+    </dependencySets>
+    <fileSets>
+        <fileSet>
+            <!-- Get the files from the top-level directory, which should be two levels above the storages -->
+            <directory>${project.basedir}/../..</directory>
+            <outputDirectory>${project.artifactId}</outputDirectory>
+            <includes>
+                <include>README*</include>
+                <include>CHANGELOG*</include>
+                <include>CONTRIBUTE*</include>
+                <include>COPYRIGHT*</include>
+                <include>LICENSE*</include>
+            </includes>
+            <useDefaultExcludes>true</useDefaultExcludes>
+        </fileSet>
+        <fileSet>
+            <directory>${project.build.directory}/generated-sources</directory>
+            <outputDirectory>${project.artifactId}</outputDirectory>
+            <includes>
+                <include>*.json</include>
+            </includes>
+            <useDefaultExcludes>true</useDefaultExcludes>
+        </fileSet>
+    </fileSets>
+</assembly>
diff --git a/debezium-bom/pom.xml b/debezium-bom/pom.xml
index 660811c9531..e5b2834b4bb 100644
--- a/debezium-bom/pom.xml
+++ b/debezium-bom/pom.xml
@@ -57,6 +57,9 @@
         <!-- K8s Config Map Storage -->
         <version.kubernetes.client>6.13.4</version.kubernetes.client>
 
+        <!-- Debezium AI -->
+        <version.langchain4j>1.0.0-beta2</version.langchain4j>
+
         <!-- Testing -->
         <version.junit>4.13.1</version.junit>
         <version.junit5>5.9.1</version.junit5>
@@ -66,7 +69,7 @@
         <version.jmh>1.21</version.jmh>
         <version.mockito>3.0.0</version.mockito>
         <version.awaitility>4.0.1</version.awaitility>
-        <version.testcontainers>1.19.1</version.testcontainers>
+        <version.testcontainers>1.20.6</version.testcontainers>
         <version.jsonpath>2.9.0</version.jsonpath>
         <version.jsonassert>1.5.0</version.jsonassert>
         <version.org.reflections>0.9.12</version.org.reflections>
@@ -556,6 +559,23 @@
                 <version>${version.kubernetes.client}</version>
             </dependency>
 
+            <!-- Debezium AI dependencies-->
+            <dependency>
+                <groupId>dev.langchain4j</groupId>
+                <artifactId>langchain4j-core</artifactId>
+                <version>${version.langchain4j}</version>
+            </dependency>
+            <dependency>
+                <groupId>dev.langchain4j</groupId>
+                <artifactId>langchain4j-embeddings-all-minilm-l6-v2</artifactId>
+                <version>${version.langchain4j}</version>
+            </dependency>
+            <dependency>
+                <groupId>dev.langchain4j</groupId>
+                <artifactId>langchain4j-ollama</artifactId>
+                <version>${version.langchain4j}</version>
+            </dependency>
+
             <!-- Testing utilities -->
             <dependency>
                 <groupId>org.testcontainers</groupId>
@@ -847,6 +867,11 @@
                 <artifactId>debezium-quarkus-outbox-deployment</artifactId>
                 <version>${project.version}</version>
             </dependency>
+            <dependency>
+                <groupId>io.debezium</groupId>
+                <artifactId>debezium-ai-embeddings</artifactId>
+                <version>${project.version}</version>
+            </dependency>
 
             <!-- Debezium test artifacts -->
             <dependency>
@@ -873,6 +898,12 @@
                 <version>${project.version}</version>
                 <type>test-jar</type>
             </dependency>
+            <dependency>
+                <groupId>io.debezium</groupId>
+                <artifactId>debezium-ai-embeddings</artifactId>
+                <version>${project.version}</version>
+                <type>test-jar</type>
+            </dependency>
         </dependencies>
     </dependencyManagement>
 </project>
diff --git a/pom.xml b/pom.xml
index a50b4e10947..c0c463e3f00 100644
--- a/pom.xml
+++ b/pom.xml
@@ -215,6 +215,7 @@
         <module>debezium-storage</module>
         <module>debezium-interceptor</module>
         <module>debezium-sink</module>
+        <module>debezium-ai</module>
     </modules>
 
     <distributionManagement>


Reviews:
user: login: vjuranek
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-26T12:37:49Z
user: login: mfvitale
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-26T13:04:09Z
user: login: mfvitale
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-26T13:18:32Z
user: login: vjuranek
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-26T13:29:42Z
user: login: mfvitale
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-26T14:05:35Z
user: login: vjuranek
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-26T14:24:26Z
user: login: jpechane
body: @vjuranek Good start, thanks! I left couple of comments.
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-27T09:55:13Z
user: login: vjuranek
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-27T10:19:47Z
user: login: jpechane
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-28T05:00:50Z
user: login: vjuranek
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-28T12:07:10Z
user: login: mfvitale
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-28T14:12:01Z
user: login: vjuranek
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-28T14:32:44Z
user: login: mfvitale
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-31T07:51:17Z
user: login: jpechane
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-01T10:54:57Z
user: login: vjuranek
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-01T12:02:24Z
user: login: vjuranek
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-01T20:33:50Z
user: login: jpechane
body: @vjuranek That's exactly what I was looking for! Thanks
There is only one thing missing - assembly descriptor similar to https://github.com/debezium/debezium/blob/main/debezium-assembly-descriptors/src/main/resources/assemblies/storage-distribution.xml

The built assamblies should not use `plugin` suffix but `embeddings` and should not contain cor, API etc. Debezium libraries, only what's needed to be copied into connector directory to get SMT running.

As we are almost there I'd really love to get it into 3.1.0.FInal as Incubating fetaure - it is omcpletely isolated so there is no risk that it breaks Debezium as such. And it will allow us to write a blogpost and example around it.

Documentation can be merged in a separate post-release PR and the announcement will just mention that the documentation is going to be published soon.
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-02T04:56:20Z

Review comments:
diff_hunk: @@ -0,0 +1,171 @@
+/*
+ * Copyright Debezium Authors.
+ *
+ * Licensed under the Apache Software License version 2.0, available at http://www.apache.org/licenses/LICENSE-2.0
+ */
+package io.debezium.ai.embeddings;
+
+import static java.lang.String.format;
+import static org.apache.kafka.connect.transforms.util.Requirements.requireStruct;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigException;
+import org.apache.kafka.connect.components.Versioned;
+import org.apache.kafka.connect.connector.ConnectRecord;
+import org.apache.kafka.connect.data.Schema;
+import org.apache.kafka.connect.data.SchemaBuilder;
+import org.apache.kafka.connect.data.Struct;
+import org.apache.kafka.connect.transforms.Transformation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import io.debezium.config.Configuration;
+import io.debezium.config.Field;
+import io.debezium.transforms.ConnectRecordUtil;
+import io.debezium.transforms.SmtManager;
+import io.debezium.util.BoundedConcurrentHashMap;
+
+import dev.langchain4j.data.embedding.Embedding;
+import dev.langchain4j.data.segment.TextSegment;
+import dev.langchain4j.model.embedding.EmbeddingModel;
+
+/**
+ * Base class of the embeddings SMT, which provides common configuration and manipulations with the Connect record.
+ * Child classes has to implement at least {@link AbstractEmbeddingsSmt#getModel()} class, which provides specific
+ * model which will be used for creating the embeddings.
+ *
+ * @author vjuranek
+ */
+public abstract class AbstractEmbeddingsSmt<R extends ConnectRecord<R>> implements Transformation<R>, Versioned {
+
+    private static final Logger LOGGER = LoggerFactory.getLogger(AbstractEmbeddingsSmt.class);
+
+    private static final Field TEXT_FIELD = Field.create("field.source")
+            .withDisplayName("Name of the record field from which embeddings should be created.")
+            .withType(ConfigDef.Type.STRING)
+            .withWidth(ConfigDef.Width.SHORT)
+            .withImportance(ConfigDef.Importance.HIGH)
+            .required()
+            .withDescription("Name of the field of the record which content will be used as an input for embeddings. Supports also nested fields.");
+
+    private static final Field EMBEDDGINS_FIELD = Field.create("field.embeddings")
+            .withDisplayName("Name of the field which would contain the embeddings of the input field.")
+            .withType(ConfigDef.Type.STRING)
+            .withWidth(ConfigDef.Width.SHORT)
+            .withImportance(ConfigDef.Importance.HIGH)
+            .required()
+            .withDescription(
+                    "Name of the field which which will be appended to the record and which would contain the embeddings of the content `filed.source` field. Supports also nested fields.");
+
+    private static final Field.Set ALL_FIELDS = Field.setOf(TEXT_FIELD, EMBEDDGINS_FIELD);
+    private static final Schema VECTOR_SCHEMA = SchemaBuilder.array(Schema.FLOAT32_SCHEMA).schema();
path: debezium-ai/debezium-ai-embeddings-base/src/main/java/io/debezium/ai/embeddings/AbstractEmbeddingsSmt.java
user: login: vjuranek
body: I wonder, if I should use `io.debezium.data.vector.FloatVector` (IMHO yes, will include it in final version).
created_at: 2025-03-26T12:37:42Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,22 @@
+/*
+ * Copyright Debezium Authors.
+ *
+ * Licensed under the Apache Software License version 2.0, available at http://www.apache.org/licenses/LICENSE-2.0
+ */
+package io.debezium.ai.embeddings;
+
+import org.apache.kafka.connect.connector.ConnectRecord;
+
+import dev.langchain4j.model.embedding.EmbeddingModel;
+import dev.langchain4j.model.embedding.onnx.allminilml6v2.AllMiniLmL6V2EmbeddingModel;
+
+/**
+ * Embeddings SMT which uses <a href="https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2">MiniLmL6V2 model</a> model for creating embeddings.
+ *
+ * @author vjuranek
+ */
+public class EmbeddingsMiniLmSmt<R extends ConnectRecord<R>> extends AbstractEmbeddingsSmt<R> {
+    public EmbeddingModel getModel() {
+        return new AllMiniLmL6V2EmbeddingModel();
path: debezium-ai/debezium-ai-embeddings-minilm/src/main/java/io/debezium/ai/embeddings/EmbeddingsMiniLmSmt.java
user: login: mfvitale
body: I'm wondering if we should use the [HuggingFace](https://docs.langchain4j.dev/integrations/embedding-models/hugging-face) directly paired with a property to choose the model. As advanced case the users can use different models with a custom STM? In that case users can easily change models without create a custom SMT. 
created_at: 2025-03-26T13:04:08Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,22 @@
+/*
+ * Copyright Debezium Authors.
+ *
+ * Licensed under the Apache Software License version 2.0, available at http://www.apache.org/licenses/LICENSE-2.0
+ */
+package io.debezium.ai.embeddings;
+
+import org.apache.kafka.connect.connector.ConnectRecord;
+
+import dev.langchain4j.model.embedding.EmbeddingModel;
+import dev.langchain4j.model.embedding.onnx.allminilml6v2.AllMiniLmL6V2EmbeddingModel;
+
+/**
+ * Embeddings SMT which uses <a href="https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2">MiniLmL6V2 model</a> model for creating embeddings.
+ *
+ * @author vjuranek
+ */
+public class EmbeddingsMiniLmSmt<R extends ConnectRecord<R>> extends AbstractEmbeddingsSmt<R> {
+    public EmbeddingModel getModel() {
+        return new AllMiniLmL6V2EmbeddingModel();
path: debezium-ai/debezium-ai-embeddings-minilm/src/main/java/io/debezium/ai/embeddings/EmbeddingsMiniLmSmt.java
user: login: mfvitale
body: The best would be if we could permit choosing from [langchain4j suported models](https://docs.langchain4j.dev/category/embedding-models) directly with a config removing the need to create a custom STM.
created_at: 2025-03-26T13:18:31Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,22 @@
+/*
+ * Copyright Debezium Authors.
+ *
+ * Licensed under the Apache Software License version 2.0, available at http://www.apache.org/licenses/LICENSE-2.0
+ */
+package io.debezium.ai.embeddings;
+
+import org.apache.kafka.connect.connector.ConnectRecord;
+
+import dev.langchain4j.model.embedding.EmbeddingModel;
+import dev.langchain4j.model.embedding.onnx.allminilml6v2.AllMiniLmL6V2EmbeddingModel;
+
+/**
+ * Embeddings SMT which uses <a href="https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2">MiniLmL6V2 model</a> model for creating embeddings.
+ *
+ * @author vjuranek
+ */
+public class EmbeddingsMiniLmSmt<R extends ConnectRecord<R>> extends AbstractEmbeddingsSmt<R> {
+    public EmbeddingModel getModel() {
+        return new AllMiniLmL6V2EmbeddingModel();
path: debezium-ai/debezium-ai-embeddings-minilm/src/main/java/io/debezium/ai/embeddings/EmbeddingsMiniLmSmt.java
user: login: vjuranek
body: I'm going to add it later on as a special module. The problematic thing is (not only for HUggingFace, but for most of the models), that you need an access token, so it's little bit hard to write tests (but more investigation in this regard is still WIP, so maybe there are some options).
Another one on the my TODO list is Ollama - here exists even [testconatiner](https://java.testcontainers.org/modules/ollama/) for it (possible issue here is size of the container and models).
created_at: 2025-03-26T13:29:42Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,22 @@
+/*
+ * Copyright Debezium Authors.
+ *
+ * Licensed under the Apache Software License version 2.0, available at http://www.apache.org/licenses/LICENSE-2.0
+ */
+package io.debezium.ai.embeddings;
+
+import org.apache.kafka.connect.connector.ConnectRecord;
+
+import dev.langchain4j.model.embedding.EmbeddingModel;
+import dev.langchain4j.model.embedding.onnx.allminilml6v2.AllMiniLmL6V2EmbeddingModel;
+
+/**
+ * Embeddings SMT which uses <a href="https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2">MiniLmL6V2 model</a> model for creating embeddings.
+ *
+ * @author vjuranek
+ */
+public class EmbeddingsMiniLmSmt<R extends ConnectRecord<R>> extends AbstractEmbeddingsSmt<R> {
+    public EmbeddingModel getModel() {
+        return new AllMiniLmL6V2EmbeddingModel();
path: debezium-ai/debezium-ai-embeddings-minilm/src/main/java/io/debezium/ai/embeddings/EmbeddingsMiniLmSmt.java
user: login: mfvitale
body: > Another one on the my TODO list is Ollama - here exists even [testconatiner](https://java.testcontainers.org/modules/ollama/) for it (possible issue here is size of the container and models).

Yes, for the test part I was thinking to it or also `LocalAI` or the one you have used (`AllMiniLmL6V2EmbeddingModel`).

I dunno what are the performances of the `AllMiniLmL6V2EmbeddingModel` if they are not comparable with the best available model I think that we are just forcing users to create a custom SMT to have good results. This is my concern. 
created_at: 2025-03-26T14:05:34Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,22 @@
+/*
+ * Copyright Debezium Authors.
+ *
+ * Licensed under the Apache Software License version 2.0, available at http://www.apache.org/licenses/LICENSE-2.0
+ */
+package io.debezium.ai.embeddings;
+
+import org.apache.kafka.connect.connector.ConnectRecord;
+
+import dev.langchain4j.model.embedding.EmbeddingModel;
+import dev.langchain4j.model.embedding.onnx.allminilml6v2.AllMiniLmL6V2EmbeddingModel;
+
+/**
+ * Embeddings SMT which uses <a href="https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2">MiniLmL6V2 model</a> model for creating embeddings.
+ *
+ * @author vjuranek
+ */
+public class EmbeddingsMiniLmSmt<R extends ConnectRecord<R>> extends AbstractEmbeddingsSmt<R> {
+    public EmbeddingModel getModel() {
+        return new AllMiniLmL6V2EmbeddingModel();
path: debezium-ai/debezium-ai-embeddings-minilm/src/main/java/io/debezium/ai/embeddings/EmbeddingsMiniLmSmt.java
user: login: vjuranek
body: I don't expect this one to be used in the production, it meant it rather as a toy SMT, which works out of the box and can be used for PoC or for testing purposes (for us , but for users as well).

But in general yes, I expect users to eventually to write their SMTs, because there are too many models and service we wont be able to cover all of them anyway, so basically my idea is to make implementation of custom SMT as easy as possible (just provide model you want to use) and implement SMTs only for few most common models/services (huggingface and ollama, eventually chatgpt - possibly based on the community input)
created_at: 2025-03-26T14:24:26Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,172 @@
+/*
+ * Copyright Debezium Authors.
+ *
+ * Licensed under the Apache Software License version 2.0, available at http://www.apache.org/licenses/LICENSE-2.0
+ */
+package io.debezium.ai.embeddings;
+
+import static java.lang.String.format;
+import static org.apache.kafka.connect.transforms.util.Requirements.requireStruct;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigException;
+import org.apache.kafka.connect.components.Versioned;
+import org.apache.kafka.connect.connector.ConnectRecord;
+import org.apache.kafka.connect.data.Schema;
+import org.apache.kafka.connect.data.Struct;
+import org.apache.kafka.connect.transforms.Transformation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import io.debezium.Module;
+import io.debezium.config.Configuration;
+import io.debezium.config.Field;
+import io.debezium.data.vector.FloatVector;
+import io.debezium.transforms.ConnectRecordUtil;
+import io.debezium.transforms.SmtManager;
+import io.debezium.util.BoundedConcurrentHashMap;
+
+import dev.langchain4j.data.embedding.Embedding;
+import dev.langchain4j.data.segment.TextSegment;
+import dev.langchain4j.model.embedding.EmbeddingModel;
+
+/**
+ * Base class of the embeddings SMT, which provides common configuration and manipulations with the Connect record.
+ * Child classes has to implement at least {@link AbstractEmbeddingsSmt#getModel()} class, which provides specific
+ * model which will be used for creating the embeddings.
+ *
+ * @author vjuranek
+ */
+public abstract class AbstractEmbeddingsSmt<R extends ConnectRecord<R>> implements Transformation<R>, Versioned {
path: debezium-ai/debezium-ai-embeddings-base/src/main/java/io/debezium/ai/embeddings/AbstractEmbeddingsSmt.java
user: login: jpechane
body: ```suggestion
public abstract class AbstractEmbeddingsTransformation<R extends ConnectRecord<R>> implements Transformation<R>, Versioned {
```
created_at: 2025-03-27T09:42:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,172 @@
+/*
+ * Copyright Debezium Authors.
+ *
+ * Licensed under the Apache Software License version 2.0, available at http://www.apache.org/licenses/LICENSE-2.0
+ */
+package io.debezium.ai.embeddings;
+
+import static java.lang.String.format;
+import static org.apache.kafka.connect.transforms.util.Requirements.requireStruct;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigException;
+import org.apache.kafka.connect.components.Versioned;
+import org.apache.kafka.connect.connector.ConnectRecord;
+import org.apache.kafka.connect.data.Schema;
+import org.apache.kafka.connect.data.Struct;
+import org.apache.kafka.connect.transforms.Transformation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import io.debezium.Module;
+import io.debezium.config.Configuration;
+import io.debezium.config.Field;
+import io.debezium.data.vector.FloatVector;
+import io.debezium.transforms.ConnectRecordUtil;
+import io.debezium.transforms.SmtManager;
+import io.debezium.util.BoundedConcurrentHashMap;
+
+import dev.langchain4j.data.embedding.Embedding;
+import dev.langchain4j.data.segment.TextSegment;
+import dev.langchain4j.model.embedding.EmbeddingModel;
+
+/**
+ * Base class of the embeddings SMT, which provides common configuration and manipulations with the Connect record.
+ * Child classes has to implement at least {@link AbstractEmbeddingsSmt#getModel()} class, which provides specific
+ * model which will be used for creating the embeddings.
+ *
+ * @author vjuranek
+ */
+public abstract class AbstractEmbeddingsSmt<R extends ConnectRecord<R>> implements Transformation<R>, Versioned {
+
+    private static final Logger LOGGER = LoggerFactory.getLogger(AbstractEmbeddingsSmt.class);
+
+    private static final Field TEXT_FIELD = Field.create("field.source")
+            .withDisplayName("Name of the record field from which embeddings should be created.")
+            .withType(ConfigDef.Type.STRING)
+            .withWidth(ConfigDef.Width.SHORT)
+            .withImportance(ConfigDef.Importance.HIGH)
+            .required()
+            .withDescription("Name of the field of the record which content will be used as an input for embeddings. Supports also nested fields.");
+
+    private static final Field EMBEDDGINS_FIELD = Field.create("field.embeddings")
path: debezium-ai/debezium-ai-embeddings-base/src/main/java/io/debezium/ai/embeddings/AbstractEmbeddingsSmt.java
user: login: jpechane
body: Can we make it optional and replace the whole payload with embeddings hwn not provided?
created_at: 2025-03-27T09:43:36Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,172 @@
+/*
+ * Copyright Debezium Authors.
+ *
+ * Licensed under the Apache Software License version 2.0, available at http://www.apache.org/licenses/LICENSE-2.0
+ */
+package io.debezium.ai.embeddings;
+
+import static java.lang.String.format;
+import static org.apache.kafka.connect.transforms.util.Requirements.requireStruct;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigException;
+import org.apache.kafka.connect.components.Versioned;
+import org.apache.kafka.connect.connector.ConnectRecord;
+import org.apache.kafka.connect.data.Schema;
+import org.apache.kafka.connect.data.Struct;
+import org.apache.kafka.connect.transforms.Transformation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import io.debezium.Module;
+import io.debezium.config.Configuration;
+import io.debezium.config.Field;
+import io.debezium.data.vector.FloatVector;
+import io.debezium.transforms.ConnectRecordUtil;
+import io.debezium.transforms.SmtManager;
+import io.debezium.util.BoundedConcurrentHashMap;
+
+import dev.langchain4j.data.embedding.Embedding;
+import dev.langchain4j.data.segment.TextSegment;
+import dev.langchain4j.model.embedding.EmbeddingModel;
+
+/**
+ * Base class of the embeddings SMT, which provides common configuration and manipulations with the Connect record.
+ * Child classes has to implement at least {@link AbstractEmbeddingsSmt#getModel()} class, which provides specific
+ * model which will be used for creating the embeddings.
+ *
+ * @author vjuranek
+ */
+public abstract class AbstractEmbeddingsSmt<R extends ConnectRecord<R>> implements Transformation<R>, Versioned {
+
+    private static final Logger LOGGER = LoggerFactory.getLogger(AbstractEmbeddingsSmt.class);
+
+    private static final Field TEXT_FIELD = Field.create("field.source")
+            .withDisplayName("Name of the record field from which embeddings should be created.")
+            .withType(ConfigDef.Type.STRING)
+            .withWidth(ConfigDef.Width.SHORT)
+            .withImportance(ConfigDef.Importance.HIGH)
+            .required()
+            .withDescription("Name of the field of the record which content will be used as an input for embeddings. Supports also nested fields.");
+
+    private static final Field EMBEDDGINS_FIELD = Field.create("field.embeddings")
+            .withDisplayName("Name of the field which would contain the embeddings of the input field.")
+            .withType(ConfigDef.Type.STRING)
+            .withWidth(ConfigDef.Width.SHORT)
+            .withImportance(ConfigDef.Importance.HIGH)
+            .required()
+            .withDescription(
+                    "Name of the field which which will be appended to the record and which would contain the embeddings of the content `filed.source` field. Supports also nested fields.");
+
+    private static final Field.Set ALL_FIELDS = Field.setOf(TEXT_FIELD, EMBEDDGINS_FIELD);
path: debezium-ai/debezium-ai-embeddings-base/src/main/java/io/debezium/ai/embeddings/AbstractEmbeddingsSmt.java
user: login: jpechane
body: ```suggestion
    public static final Field.Set ALL_FIELDS = Field.setOf(TEXT_FIELD, EMBEDDGINS_FIELD);
```
created_at: 2025-03-27T09:44:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,172 @@
+/*
+ * Copyright Debezium Authors.
+ *
+ * Licensed under the Apache Software License version 2.0, available at http://www.apache.org/licenses/LICENSE-2.0
+ */
+package io.debezium.ai.embeddings;
+
+import static java.lang.String.format;
+import static org.apache.kafka.connect.transforms.util.Requirements.requireStruct;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigException;
+import org.apache.kafka.connect.components.Versioned;
+import org.apache.kafka.connect.connector.ConnectRecord;
+import org.apache.kafka.connect.data.Schema;
+import org.apache.kafka.connect.data.Struct;
+import org.apache.kafka.connect.transforms.Transformation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import io.debezium.Module;
+import io.debezium.config.Configuration;
+import io.debezium.config.Field;
+import io.debezium.data.vector.FloatVector;
+import io.debezium.transforms.ConnectRecordUtil;
+import io.debezium.transforms.SmtManager;
+import io.debezium.util.BoundedConcurrentHashMap;
+
+import dev.langchain4j.data.embedding.Embedding;
+import dev.langchain4j.data.segment.TextSegment;
+import dev.langchain4j.model.embedding.EmbeddingModel;
+
+/**
+ * Base class of the embeddings SMT, which provides common configuration and manipulations with the Connect record.
+ * Child classes has to implement at least {@link AbstractEmbeddingsSmt#getModel()} class, which provides specific
+ * model which will be used for creating the embeddings.
+ *
+ * @author vjuranek
+ */
+public abstract class AbstractEmbeddingsSmt<R extends ConnectRecord<R>> implements Transformation<R>, Versioned {
+
+    private static final Logger LOGGER = LoggerFactory.getLogger(AbstractEmbeddingsSmt.class);
+
+    private static final Field TEXT_FIELD = Field.create("field.source")
+            .withDisplayName("Name of the record field from which embeddings should be created.")
+            .withType(ConfigDef.Type.STRING)
+            .withWidth(ConfigDef.Width.SHORT)
+            .withImportance(ConfigDef.Importance.HIGH)
+            .required()
+            .withDescription("Name of the field of the record which content will be used as an input for embeddings. Supports also nested fields.");
+
+    private static final Field EMBEDDGINS_FIELD = Field.create("field.embeddings")
+            .withDisplayName("Name of the field which would contain the embeddings of the input field.")
+            .withType(ConfigDef.Type.STRING)
+            .withWidth(ConfigDef.Width.SHORT)
+            .withImportance(ConfigDef.Importance.HIGH)
+            .required()
+            .withDescription(
+                    "Name of the field which which will be appended to the record and which would contain the embeddings of the content `filed.source` field. Supports also nested fields.");
+
+    private static final Field.Set ALL_FIELDS = Field.setOf(TEXT_FIELD, EMBEDDGINS_FIELD);
+    private static final Schema VECTOR_SCHEMA = FloatVector.schema();
path: debezium-ai/debezium-ai-embeddings-base/src/main/java/io/debezium/ai/embeddings/AbstractEmbeddingsSmt.java
user: login: jpechane
body: ```suggestion
    private static final Schema EMBEDDING_SCHEMA = FloatVector.schema();
```
created_at: 2025-03-27T09:46:51Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,22 @@
+/*
+ * Copyright Debezium Authors.
+ *
+ * Licensed under the Apache Software License version 2.0, available at http://www.apache.org/licenses/LICENSE-2.0
+ */
+package io.debezium.ai.embeddings;
+
+import org.apache.kafka.connect.connector.ConnectRecord;
+
+import dev.langchain4j.model.embedding.EmbeddingModel;
+import dev.langchain4j.model.embedding.onnx.allminilml6v2.AllMiniLmL6V2EmbeddingModel;
+
+/**
+ * Embeddings SMT which uses <a href="https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2">MiniLmL6V2 model</a> model for creating embeddings.
+ *
+ * @author vjuranek
+ */
+public class EmbeddingsMiniLmSmt<R extends ConnectRecord<R>> extends AbstractEmbeddingsSmt<R> {
path: debezium-ai/debezium-ai-embeddings-minilm/src/main/java/io/debezium/ai/embeddings/EmbeddingsMiniLmSmt.java
user: login: jpechane
body: Do we need to implement it as another PR? Can't we just have a model factory class that will receive configuration options from the transformation and provide model? Thus there will be only single transformation and it will just have a config options choosing the right factory. I guess this is in-line with @mfvitale comments.
created_at: 2025-03-27T09:52:36Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,22 @@
+/*
+ * Copyright Debezium Authors.
+ *
+ * Licensed under the Apache Software License version 2.0, available at http://www.apache.org/licenses/LICENSE-2.0
+ */
+package io.debezium.ai.embeddings;
+
+import org.apache.kafka.connect.connector.ConnectRecord;
+
+import dev.langchain4j.model.embedding.EmbeddingModel;
+import dev.langchain4j.model.embedding.onnx.allminilml6v2.AllMiniLmL6V2EmbeddingModel;
+
+/**
+ * Embeddings SMT which uses <a href="https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2">MiniLmL6V2 model</a> model for creating embeddings.
+ *
+ * @author vjuranek
+ */
+public class EmbeddingsMiniLmSmt<R extends ConnectRecord<R>> extends AbstractEmbeddingsSmt<R> {
+    public EmbeddingModel getModel() {
+        return new AllMiniLmL6V2EmbeddingModel();
path: debezium-ai/debezium-ai-embeddings-minilm/src/main/java/io/debezium/ai/embeddings/EmbeddingsMiniLmSmt.java
user: login: jpechane
body: @vjuranek I agree that we should provide just a subset of models, IMHO this set would make the most sense

- huggingface
- ollama
- chatgpt
- claude

We can let community to provide the others.
created_at: 2025-03-27T09:54:15Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,22 @@
+/*
+ * Copyright Debezium Authors.
+ *
+ * Licensed under the Apache Software License version 2.0, available at http://www.apache.org/licenses/LICENSE-2.0
+ */
+package io.debezium.ai.embeddings;
+
+import org.apache.kafka.connect.connector.ConnectRecord;
+
+import dev.langchain4j.model.embedding.EmbeddingModel;
+import dev.langchain4j.model.embedding.onnx.allminilml6v2.AllMiniLmL6V2EmbeddingModel;
+
+/**
+ * Embeddings SMT which uses <a href="https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2">MiniLmL6V2 model</a> model for creating embeddings.
+ *
+ * @author vjuranek
+ */
+public class EmbeddingsMiniLmSmt<R extends ConnectRecord<R>> extends AbstractEmbeddingsSmt<R> {
path: debezium-ai/debezium-ai-embeddings-minilm/src/main/java/io/debezium/ai/embeddings/EmbeddingsMiniLmSmt.java
user: login: vjuranek
body: This id definitely doable, but I think the main issue with this approach is that you would have to include all the dependencies into a single jar file, which may be eventually pretty large. I guess this is the reason why langchain4j (and other project like e.g. [Quarkus](https://github.com/quarkiverse/quarkus-langchain4j/blob/main/model-providers/pom.xml#L13-L30)) separates each model into separate module and build a dedicated artifact for each model.
created_at: 2025-03-27T10:17:40Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,172 @@
+/*
+ * Copyright Debezium Authors.
+ *
+ * Licensed under the Apache Software License version 2.0, available at http://www.apache.org/licenses/LICENSE-2.0
+ */
+package io.debezium.ai.embeddings;
+
+import static java.lang.String.format;
+import static org.apache.kafka.connect.transforms.util.Requirements.requireStruct;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.kafka.common.config.ConfigDef;
+import org.apache.kafka.common.config.ConfigException;
+import org.apache.kafka.connect.components.Versioned;
+import org.apache.kafka.connect.connector.ConnectRecord;
+import org.apache.kafka.connect.data.Schema;
+import org.apache.kafka.connect.data.Struct;
+import org.apache.kafka.connect.transforms.Transformation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import io.debezium.Module;
+import io.debezium.config.Configuration;
+import io.debezium.config.Field;
+import io.debezium.data.vector.FloatVector;
+import io.debezium.transforms.ConnectRecordUtil;
+import io.debezium.transforms.SmtManager;
+import io.debezium.util.BoundedConcurrentHashMap;
+
+import dev.langchain4j.data.embedding.Embedding;
+import dev.langchain4j.data.segment.TextSegment;
+import dev.langchain4j.model.embedding.EmbeddingModel;
+
+/**
+ * Base class of the embeddings SMT, which provides common configuration and manipulations with the Connect record.
+ * Child classes has to implement at least {@link AbstractEmbeddingsSmt#getModel()} class, which provides specific
+ * model which will be used for creating the embeddings.
+ *
+ * @author vjuranek
+ */
+public abstract class AbstractEmbeddingsSmt<R extends ConnectRecord<R>> implements Transformation<R>, Versioned {
+
+    private static final Logger LOGGER = LoggerFactory.getLogger(AbstractEmbeddingsSmt.class);
+
+    private static final Field TEXT_FIELD = Field.create("field.source")
+            .withDisplayName("Name of the record field from which embeddings should be created.")
+            .withType(ConfigDef.Type.STRING)
+            .withWidth(ConfigDef.Width.SHORT)
+            .withImportance(ConfigDef.Importance.HIGH)
+            .required()
+            .withDescription("Name of the field of the record which content will be used as an input for embeddings. Supports also nested fields.");
+
+    private static final Field EMBEDDGINS_FIELD = Field.create("field.embeddings")
path: debezium-ai/debezium-ai-embeddings-base/src/main/java/io/debezium/ai/embeddings/AbstractEmbeddingsSmt.java
user: login: vjuranek
body: sure
created_at: 2025-03-27T10:18:47Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,22 @@
+/*
+ * Copyright Debezium Authors.
+ *
+ * Licensed under the Apache Software License version 2.0, available at http://www.apache.org/licenses/LICENSE-2.0
+ */
+package io.debezium.ai.embeddings;
+
+import org.apache.kafka.connect.connector.ConnectRecord;
+
+import dev.langchain4j.model.embedding.EmbeddingModel;
+import dev.langchain4j.model.embedding.onnx.allminilml6v2.AllMiniLmL6V2EmbeddingModel;
+
+/**
+ * Embeddings SMT which uses <a href="https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2">MiniLmL6V2 model</a> model for creating embeddings.
+ *
+ * @author vjuranek
+ */
+public class EmbeddingsMiniLmSmt<R extends ConnectRecord<R>> extends AbstractEmbeddingsSmt<R> {
path: debezium-ai/debezium-ai-embeddings-minilm/src/main/java/io/debezium/ai/embeddings/EmbeddingsMiniLmSmt.java
user: login: jpechane
body: @vjuranek You don't need to have it all in the single JAR. Every embeddings provider can be still in a separate module and there can be distinct `tar.gz` files for each provider, in the same way we do for storage modules. The user will then download and install the one he want to use.
created_at: 2025-03-28T05:00:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,22 @@
+/*
+ * Copyright Debezium Authors.
+ *
+ * Licensed under the Apache Software License version 2.0, available at http://www.apache.org/licenses/LICENSE-2.0
+ */
+package io.debezium.ai.embeddings;
+
+import org.apache.kafka.connect.connector.ConnectRecord;
+
+import dev.langchain4j.model.embedding.EmbeddingModel;
+import dev.langchain4j.model.embedding.onnx.allminilml6v2.AllMiniLmL6V2EmbeddingModel;
+
+/**
+ * Embeddings SMT which uses <a href="https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2">MiniLmL6V2 model</a> model for creating embeddings.
+ *
+ * @author vjuranek
+ */
+public class EmbeddingsMiniLmSmt<R extends ConnectRecord<R>> extends AbstractEmbeddingsSmt<R> {
path: debezium-ai/debezium-ai-embeddings-minilm/src/main/java/io/debezium/ai/embeddings/EmbeddingsMiniLmSmt.java
user: login: vjuranek
body: Sorry, I'm not sure I follow, or better to say, what is the difference with this approach.
So basically you are suggesting instead of 
```
transforms: embeddings,
transforms.embeddings.type: io.debezium.ai.embeddings.HuggingFaceEmbeddings,
transforms.embeddings.token: ...
...
```
use something like
```
transforms: embeddings,
transforms.embeddings.type: io.debezium.ai.embeddings.AppendEmbeddings,
transforms.embeddings.modelProvider: dev.langchain4j.model.huggingface.HuggingFaceEmbeddingModel
transforms.embeddings.token: ...
...
```
I'd say the first option is more easy and straightforward.

Or, if it was meant to introduce some general map-base configuration, load dynamically constructor and pass this map base config to if, I'm afraid it's not possible, as langchain4j doesn't offer anything like this AFAICT and moreover each model is constructed in a little bit different way (e.g. Hugging face constructs it's own HTTP client itself, while e.g. in case of Ollama you have to provide instance of the HTTP client to the constructor and for other models it's even more complicated), so you would end up constructing given model based on the config in each sub-module and that's basically what I'm suggesting with abstract `getModel()` method.

Or, if I'm completely missing your point, please elaborate more how you envision it.
created_at: 2025-03-28T12:07:10Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,22 @@
+/*
+ * Copyright Debezium Authors.
+ *
+ * Licensed under the Apache Software License version 2.0, available at http://www.apache.org/licenses/LICENSE-2.0
+ */
+package io.debezium.ai.embeddings;
+
+import org.apache.kafka.connect.connector.ConnectRecord;
+
+import dev.langchain4j.model.embedding.EmbeddingModel;
+import dev.langchain4j.model.embedding.onnx.allminilml6v2.AllMiniLmL6V2EmbeddingModel;
+
+/**
+ * Embeddings SMT which uses <a href="https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2">MiniLmL6V2 model</a> model for creating embeddings.
+ *
+ * @author vjuranek
+ */
+public class EmbeddingsMiniLmSmt<R extends ConnectRecord<R>> extends AbstractEmbeddingsSmt<R> {
path: debezium-ai/debezium-ai-embeddings-minilm/src/main/java/io/debezium/ai/embeddings/EmbeddingsMiniLmSmt.java
user: login: mfvitale
body: I think that Jiri mean to have a separate maven module for each supported langchain4j model. So you will have something like this:

* debezium-ai-embeddings-allminiLmL6V2E
* debezium-ai-embeddings-huggingface
* debezium-ai-embeddings-ollama
* debezium-ai-embeddings-chatgpt
* debezium-ai-embeddings-claude

In that way each module as it dedicated model dependency and so the users don't need to use them all. 
created_at: 2025-03-28T14:12:01Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,22 @@
+/*
+ * Copyright Debezium Authors.
+ *
+ * Licensed under the Apache Software License version 2.0, available at http://www.apache.org/licenses/LICENSE-2.0
+ */
+package io.debezium.ai.embeddings;
+
+import org.apache.kafka.connect.connector.ConnectRecord;
+
+import dev.langchain4j.model.embedding.EmbeddingModel;
+import dev.langchain4j.model.embedding.onnx.allminilml6v2.AllMiniLmL6V2EmbeddingModel;
+
+/**
+ * Embeddings SMT which uses <a href="https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2">MiniLmL6V2 model</a> model for creating embeddings.
+ *
+ * @author vjuranek
+ */
+public class EmbeddingsMiniLmSmt<R extends ConnectRecord<R>> extends AbstractEmbeddingsSmt<R> {
path: debezium-ai/debezium-ai-embeddings-minilm/src/main/java/io/debezium/ai/embeddings/EmbeddingsMiniLmSmt.java
user: login: vjuranek
body: Well, this is exactly what I have now and what I mentioned at the very beginning (`I'm going to add it later on as a special module.`). So this is either misunderstanding and we are in agreement or I'm still missing what is your preferred way how to implement it.
created_at: 2025-03-28T14:32:44Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,22 @@
+/*
+ * Copyright Debezium Authors.
+ *
+ * Licensed under the Apache Software License version 2.0, available at http://www.apache.org/licenses/LICENSE-2.0
+ */
+package io.debezium.ai.embeddings;
+
+import org.apache.kafka.connect.connector.ConnectRecord;
+
+import dev.langchain4j.model.embedding.EmbeddingModel;
+import dev.langchain4j.model.embedding.onnx.allminilml6v2.AllMiniLmL6V2EmbeddingModel;
+
+/**
+ * Embeddings SMT which uses <a href="https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2">MiniLmL6V2 model</a> model for creating embeddings.
+ *
+ * @author vjuranek
+ */
+public class EmbeddingsMiniLmSmt<R extends ConnectRecord<R>> extends AbstractEmbeddingsSmt<R> {
path: debezium-ai/debezium-ai-embeddings-minilm/src/main/java/io/debezium/ai/embeddings/EmbeddingsMiniLmSmt.java
user: login: mfvitale
body: Well my initial idea was to have all in a unique module but then you highlight the problem of dimensions and so I lead to what was your initial proposal. 

Another approach could be to have the `factory` approach proposed by Jiri and just add dependency as provided so the user then need to provide it. This approach could simplify the AI module structure. 

Honestly don't have strong opinion. The important thing is, for the users, to have an out-of-the-box way to use different models without going to customize. 
created_at: 2025-03-31T07:51:17Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,22 @@
+/*
+ * Copyright Debezium Authors.
+ *
+ * Licensed under the Apache Software License version 2.0, available at http://www.apache.org/licenses/LICENSE-2.0
+ */
+package io.debezium.ai.embeddings;
+
+import org.apache.kafka.connect.connector.ConnectRecord;
+
+import dev.langchain4j.model.embedding.EmbeddingModel;
+import dev.langchain4j.model.embedding.onnx.allminilml6v2.AllMiniLmL6V2EmbeddingModel;
+
+/**
+ * Embeddings SMT which uses <a href="https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2">MiniLmL6V2 model</a> model for creating embeddings.
+ *
+ * @author vjuranek
+ */
+public class EmbeddingsMiniLmSmt<R extends ConnectRecord<R>> extends AbstractEmbeddingsSmt<R> {
path: debezium-ai/debezium-ai-embeddings-minilm/src/main/java/io/debezium/ai/embeddings/EmbeddingsMiniLmSmt.java
user: login: jpechane
body: @vjuranek I think our only disagreemnt is if there should be multiple SMTs - each in a separate module. Or a single SMT that will just just use an embedding provided by module on a classpath.
I'd prefer the latter but it might make sense to sit on a call and discuss pros and cons of each of the options.
created_at: 2025-04-01T10:54:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,22 @@
+/*
+ * Copyright Debezium Authors.
+ *
+ * Licensed under the Apache Software License version 2.0, available at http://www.apache.org/licenses/LICENSE-2.0
+ */
+package io.debezium.ai.embeddings;
+
+import org.apache.kafka.connect.connector.ConnectRecord;
+
+import dev.langchain4j.model.embedding.EmbeddingModel;
+import dev.langchain4j.model.embedding.onnx.allminilml6v2.AllMiniLmL6V2EmbeddingModel;
+
+/**
+ * Embeddings SMT which uses <a href="https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2">MiniLmL6V2 model</a> model for creating embeddings.
+ *
+ * @author vjuranek
+ */
+public class EmbeddingsMiniLmSmt<R extends ConnectRecord<R>> extends AbstractEmbeddingsSmt<R> {
path: debezium-ai/debezium-ai-embeddings-minilm/src/main/java/io/debezium/ai/embeddings/EmbeddingsMiniLmSmt.java
user: login: vjuranek
body: @jpechane ok, I think I finally understand what/how you meant:-))... and agree it's a better approach:-)
I'll rework the PR to reflect it.
created_at: 2025-04-01T12:02:24Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,22 @@
+/*
+ * Copyright Debezium Authors.
+ *
+ * Licensed under the Apache Software License version 2.0, available at http://www.apache.org/licenses/LICENSE-2.0
+ */
+package io.debezium.ai.embeddings;
+
+import org.apache.kafka.connect.connector.ConnectRecord;
+
+import dev.langchain4j.model.embedding.EmbeddingModel;
+import dev.langchain4j.model.embedding.onnx.allminilml6v2.AllMiniLmL6V2EmbeddingModel;
+
+/**
+ * Embeddings SMT which uses <a href="https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2">MiniLmL6V2 model</a> model for creating embeddings.
+ *
+ * @author vjuranek
+ */
+public class EmbeddingsMiniLmSmt<R extends ConnectRecord<R>> extends AbstractEmbeddingsSmt<R> {
path: debezium-ai/debezium-ai-embeddings-minilm/src/main/java/io/debezium/ai/embeddings/EmbeddingsMiniLmSmt.java
user: login: vjuranek
body: @jpechane @mfvitale done, please review if is this the direction you intended - there is only a single SMT `FieldToEmbedding` in `debezium-ai-embeddings` module, which relies on model factory which delivers concrete implementation of the model used for creating embeddings. Model factory is loaded via SPI and other modules contain implementations of this model factory.
created_at: 2025-04-01T20:33:50Z
author_association: MEMBER

PR comments:
user: login: mfvitale
created_at: 2025-04-02T06:42:38Z
body: @vjuranek great work!
author_association: MEMBER
user: login: vjuranek
created_at: 2025-04-02T09:28:33Z
body: @jpechane done. I removed from the bits also `debezium-ai-embeddings` which contains SMT itself. So you have to download  `debezium-ai-embeddings-3.1.0-SNAPSHOT-embeddings.tar.gz` and e.g. `debezium-ai-embeddings-ollama-3.1.0-SNAPSHOT-embeddings.tar.gz`. If you want to have  `debezium-ai-embeddings` in all the bits so you would need only to download and deploy e.g. `debezium-ai-embeddings-ollama-3.1.0-SNAPSHOT-embeddings.tar.gz`, let me know.
author_association: MEMBER
user: login: jpechane
created_at: 2025-04-02T09:48:25Z
body: @vjuranek Yes, I'd like to have embedding provider granularity. So the `debezium-ai-embeddings` should be included in ollama. So at the end there will be only two archives one for ollama and one for minilm. There should be no assambly in `debezium-ai-embeddings` module .
author_association: CONTRIBUTOR
user: login: vjuranek
created_at: 2025-04-02T09:54:57Z
body: @jpechane done
author_association: MEMBER
user: login: jpechane
created_at: 2025-04-02T10:10:57Z
body: @vjuranek And applied, huge thanks!
BTW it is funny if you compare size of ollama and minilm distrost and then try to fgure out what is mini about mini :-)
author_association: CONTRIBUTOR