Repository: kubeflow/blog
Pull request #159: Synthetic data generation using SDV and KFP 
Author: login: tarekabouzeid
Labels: name: lgtm
name: approved
name: size/L
Created: 2025-02-16T17:27:07Z | Merged: 2025-03-03T10:12:51Z

AI-Generated Synthetic Data, using Kubeflow pipelines blog.
Authored by @akeed and @tarekabouzeid .


Diff:
diff --git a/_posts/2025-02-16-synthetic-data-using-kfp.md b/_posts/2025-02-16-synthetic-data-using-kfp.md
new file mode 100755
index 00000000..024f0f12
--- /dev/null
+++ b/_posts/2025-02-16-synthetic-data-using-kfp.md
@@ -0,0 +1,217 @@
+---
+title: "Synthetic Data Generation with Kubeflow Pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+### Synthetic Data Generation - Why and How?
+
+When creating insights, decisions, and actions from data, the best results come from real data. But accessing real data often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete. 
+
+To address these challenges, synthetic data provides an alternative. It mimics real data’s statistical properties while preserving privacy and accessibility. Synthetic data generators (synthesizers) are models trained on real data to generate new datasets that follow the same statistical distributions and relationships but do not contain real records. This allows for accelerated development, improved data availability, and enhanced privacy.
+
+Depending on the technique used, synthetic data not only mirrors statistical base properties of real data but also preserves correlations between features. These synthesizers — such as those based on Gaussian Copulas, Generative Adversarial Networks (GANs), and Variational Autoencoders (VAEs) — enable the creation of high-fidelity synthetic datasets. See more description of these techniques below.
+
+### Key Benefits of Using Synthetic Data
+
+While the above focuses on speed of development in general, and augmentation of data to improve performance of analytical modes, there are more motivations for *creating* (synthetic) data:
+
+- **Enhanced Privacy and Security**  
+  Mimics real datasets without containing sensitive or personally identifiable information, mitigating privacy risks and ensuring compliance with regulations like GDPR.
+
+- **Improved Data Availability**  
+  Enables testing and training of models without requiring extensive real-world data collection.  
+
+- **Innovation and Experimentation**  
+  Allows safe experimentation with new algorithms and models without exposing sensitive data, fostering rapid prototyping in a secure environment.  
+
+- **Ethical and Responsible AI Development**  
+  Ensures training data is free from biases present in real-world datasets, promoting fair and unbiased AI systems.  
+
+- **Accelerated Testing and Deployment**  
+  Supports testing of new products, services, and systems in a controlled yet realistic setting, ensuring they are robust, scalable, and ready for real-world use.  
+
+- **Cost Efficiency**  
+  Reduces expenses related to data collection, storage, and compliance by eliminating the need for large-scale real-world data acquisition.  
+
+- **Regulatory Compliance Simplification**  
+  Helps organizations navigate complex data regulations by offering a compliant alternative to real-world datasets, easing cross-border data transfers.  
+
+- **Balanced and Augmented Datasets**  
+  Supplements real-world data by balancing underrepresented classes, improving model performance, and reducing biases in AI training.  
+
+- **Resilience Against Data Scarcity**  
+  Enables AI development in domains where real-world data is limited, expensive, or difficult to obtain—such as healthcare and cybersecurity—by generating high-quality alternative datasets.
+
+To realize these benefits, we need effective tools for generating synthetic data. Different frameworks exist for this purpose, ranging from cloud-based platforms to open-source solutions. In this post, we focus on **open-source synthetic data generation frameworks** that provide control, flexibility, and on-premise deployment options.
+
+### Frameworks for Creating Synthetic Data 
+
+This post focuses exclusively on open source frameworks.
+Some data cannot be sent to the cloud, so some cloud-based synthetic data generation solutions are not always a good fit. 
+For data already in cloud, we can use other cloud-based frameworks to generate synthetic data.
+
+Synthesizers are motivated by multiple factors, but in this context, our focus remains on generating synthetic data for on-premise use.
+
+So, what framework did we (initially) choose? Currently, we are using the open source version of [SDV](https://sdv.dev/), 
+an easy-to-use framework with a strong community and many useful features out-of-the-box (e.g. built-in evaluators, many modeling techniques). 
+The field of synthetic data is evolving rapidly. While we do not aim to cover the latest advancements exhaustively, the use of Foundation models is certainly an area of interest.
+
+One of the most widely used open-source libraries for synthetic data generation is **Synthetic Data Vault (SDV)**. It provides multiple synthesizers, each tailored for different types of data and statistical properties.
+
+### The Synthetic Data Vault (SDV)
+
+When you initialize and fit a synthesizer (like GaussianCopulaSynthesizer, CTGANSynthesizer, etc. - see below), it trains a model based on 
+the dataset you provide. This model learns the distribution of the data, capturing the relationships and dependencies between 
+different features in the dataset. The synthesizer doesn't memorize individual records from the dataset. Instead, it tries to learn the underlying statistical patterns, correlations, and distributions present in the data. 
+
+Below are the (free) synthesizers provided by SDV that we evaluated on each use case. Each synthesizer does this differently:
+
+- **GaussianCopulaSynthesizer:** Uses statistical copula functions to model relationships between features, ensuring accurate marginal distributions.
+- **CTGANSynthesizer:** Uses Generative Adversarial Networks (GANs) to learn complex data distributions, particularly effective for categorical and mixed-type data.
+- **TVAESynthesizer:** Leverages Variational Autoencoders (VAEs) to capture latent representations, useful for continuous and structured data.
+- **CopulaGANSynthesizer:** Combines Copula-based statistical modeling with GANs to generate data with complex dependencies.
+- **PARSynthesizer:** Uses autoregressive models to generate sequential data while preserving temporal dependencies.
+
+*There are more synthesizers, also from SDV, but not all are open source.* We used the first four, when evaluating optimal synthesizer for our different use cases.
+
+**Generators - generating new data - on demand**
+
+Synthesizers are statistical and (more often) AI models trained to mimic the real data. Once developed, the resulting models are used to create as much synthetic data as you find useful for your use case. Once trained, the synthesizer uses the learned model to generate new synthetic data that follows the same statistical properties and distributions as the original dataset, without directly copying any real data points. If you need more data? Just call the generator.
+
+### Evaluation Criteria for Synthetic Data
+
+But, how good is synthetic data, how do we evaluate it?
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthetic data generation technique (synthesizer) is best for our specific dataset and use case.  
+
+We need to ensure a good balance between: 
+
+- Usability – How useful is the synthetic data for the intended use case?
+- Fidelity – How well does the synthetic data preserve statistical properties of the real data?
+- Privacy – Does the generated data ensure an acceptable level of privacy for the given use case?
+
+For now, we are focusing only on usability and fidelity, using framework-provided measurements for fidelity and workflows described below to assess usability.
+
+**Comments on privacy and privacy preserving techniques**
+
+Ensuring privacy in synthetic data is a non-trivial problem, even if there are techniques to ensure levels of privacy, it remains an active area of research. 
+
+*Privacy problems, in synthetic data?*
+
+While synthetic data enhances privacy by removing personally identifiable information, it is not inherently risk-free. Some key challenges include:
+
+- Overfitting and Memorization: If a synthesizer is overfitted, it may generate synthetic records that closely resemble real data, leading to privacy leakage.
+- Anomaly Exposure: Unique individuals or rare events in the dataset (e.g., a very wealthy individual or a rare disease) may be unintentionally replicated in synthetic data, creating a risk of re-identification.
+- Re-identification Attacks: Even if synthetic data is statistically different from real data, attackers may use background knowledge to infer sensitive details about individuals.
+
+One additional problem here is that it might be the anomalies we really are looking for. Currently we are experimenting with various differential privacy strategies, but it is still early days, and we do not focus on them in the examples below.
+
+### Our On-Premise Analytics Platform: ARCUS
+
+<div style="display: flex; align-items: center; gap: 20px;">
+  <p>
+    ARCUS is Telia’s advanced on-premise analytics platform, designed to support a wide range of use cases.
+    The platform provides a Kubeflow-based MLOps environment for descriptive, predictive, generative, and (ongoing) agentic AI. 
+    Fully built on open-source, ARCUS integrates a comprehensive stack of components into a unified platform - where Kubernetes is the cornerstone.
+  </p>
+</div>
+
+### Needed environment to create synthetic data
+
+For an efficient, automated selection of the best synthesizer, we need a number of things - from the underlying platform with GPUs and MLOps (Kubeflow).
+
+- Kubeflow pipelines
+- GPU capabilities (for performance and efficiency)
+- Development (IDE) environment (for framework building and running)
+- Modern data platform (MinIO, Airflow) automating the synthetic data generation datasets
+
+#### Parallelism needed
+
+In the (Kube)flows below, we run evaluations in parallel - one for respective synthesizer, followed by a comparison of usability and fidelity scores, selecting the 'winner'.
+
+*Note:* In earlier version of Kubeflow we noticed that the parallelism wasn't acting as expected, waiting for all threads to complete before moving to next step. We had to create a temporary workaround for this, now solved in Kubeflow Pipelines 2.3.0.
+
+Below, we briefly describe the base flow for selecting synthesizer, followed by one use case where we use the resulting data generator for ML development in cloud.
+
+## Exploring the Creation and Usefulness of Synthetic Data
+
+This is what we want to do: we have a use case, the supporting data, and developed ML model.
+
+- How similar is the synthetic data compared to the real data (interesting for e.g. visualization use cases)?
+- How well do the ML models based on synthetic data keep up with ML models based on real data?
+
+**Validation of synthetic data techniques**
+
+1. Create the synthetic data and save the best synthetic data generator. In this step similarity measures are created by the out of the box SDV framework
+2. Create the ML model (in our case classifier model) both on real data and the using the synthetic data. Compare the performance of both models against the same real data testset.
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image-2.png" width="800">
+
+From above, we have an example where the final synthesizer is collected and saved. This step is used in the example  below, exporting the resulting synthetic data generator to cloud.
+
+## Using Synthetic Data Generators to Enable Multiple Environments without Data Transfer
+
+*Below is a usecase where we need to make use of both on-premise and cloud, without moving data to cloud.*
+
+**Problem statement:** 
+
+- Our data cannot be moved from on-premise to cloud.
+- We need extra compute power, in our public cloud environment, to create an ML model for use on-premise.
+- The ML model is to be used on-premise, on new incoming data streams (that cannot be moved to cloud)
+
+**Solution:**
+
+1. Create synthetic data for our on-premise environment use-cases, and - as a side product we save 
+away the synthetic data generator (the pickled model used to create synthetic data). 
+2. Copy the synthetic data generator to cloud
+3. Use the synthetic data generator in the cloud, creating synthetic data for training of an ML model
+4. Copy the ML model on-premise, and use it for new incoming data
+5. Evaluate: Compare the on-premise AI model with the model created in the cloud - against the same test data
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image.png" width="700">
+
+
+**Division of work, what is done on-premise with Kubeflow, and what is done in cloud (AWS SageMaker)?**
+
+##### On-premise
+
+See the above *Validation of synthetic data techniques*.
+
+- Develop the model on real data – for the comparison later with the cloud model.
+- Create synthetic generators, evaluate the generators, and export the best generator to AWS.
+
+##### Cloud
+
+- Use the imported synthetic generator (from on-premise)
+- Create synthetic data using the synthetic data generator
+- Develop the model and determine which synthetic generator is the best
+- Increase the amount of synthetic data, to see if the increase of synthetic data improves model performance (not for sure it will, see below comment)
+- Export model to on-premise
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image-3.png" width="500">
+
+
+In some more detail below.
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image-4.png" width="700">
+
+##### On-premise
+
+- Compare real data model against synthetic data model – using real test data.
+
+*In the current examples we see near equivalent performance of the ML models (a few percentage points lower for models created using synthetic data). We experimented with increasing the size of the synthetic dataset, with minor improvements. Augmenting the training data is expected (not tested here) to have more effects when using deep learning algorithms.*
+
+# Summary
+
+Clearly, the above workflows would be very cumbersome to build and maintain without Kubeflow. 
+Our solution is entirely open source, Kubernetes based, and uses Kubeflow and SDV to give us the scalability, robustness, and detailed control that is required.
+
+The area of synthetic data generation is moving fast with the overall AI field. 
+Reports from [IBM](https://www.ibm.com/think/topics/synthetic-data) and others, of the increased usage of synthetic data for e.g. LLM training is frequent but the application areas are much greater.
+We also expect more capable synthesizers and, hopefully, privacy preserving techniques to keep up with the innovation in this area.
+Our original main motivator was speed up in innovation and experimentation, and overall - speed to market. Often a key pain for our teams.
+
+Looking ahead, we are exploring the development of a synthesizer catalog — ideally integrated into our overall data catalog — to enable users to rapidly experiment with ideas and get started more efficiently.
diff --git a/images/2025-02-16-synthetic-data-using-kfp/image-2.png b/images/2025-02-16-synthetic-data-using-kfp/image-2.png
new file mode 100755
index 00000000..afc0411c
Binary files /dev/null and b/images/2025-02-16-synthetic-data-using-kfp/image-2.png differ
diff --git a/images/2025-02-16-synthetic-data-using-kfp/image-3.png b/images/2025-02-16-synthetic-data-using-kfp/image-3.png
new file mode 100755
index 00000000..4e8cb183
Binary files /dev/null and b/images/2025-02-16-synthetic-data-using-kfp/image-3.png differ
diff --git a/images/2025-02-16-synthetic-data-using-kfp/image-4.png b/images/2025-02-16-synthetic-data-using-kfp/image-4.png
new file mode 100755
index 00000000..f3651329
Binary files /dev/null and b/images/2025-02-16-synthetic-data-using-kfp/image-4.png differ
diff --git a/images/2025-02-16-synthetic-data-using-kfp/image.png b/images/2025-02-16-synthetic-data-using-kfp/image.png
new file mode 100755
index 00000000..fea7b203
Binary files /dev/null and b/images/2025-02-16-synthetic-data-using-kfp/image.png differ


Reviews:
user: login: franciscojavierarceo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-24T16:32:00Z
user: login: franciscojavierarceo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-24T16:32:25Z
user: login: franciscojavierarceo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-24T16:34:05Z
user: login: franciscojavierarceo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-24T16:35:09Z
user: login: franciscojavierarceo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-24T16:35:55Z
user: login: franciscojavierarceo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-24T16:38:41Z
user: login: franciscojavierarceo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-24T16:38:59Z
user: login: franciscojavierarceo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-24T16:39:51Z
user: login: franciscojavierarceo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-24T16:40:15Z
user: login: franciscojavierarceo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-24T16:42:07Z
user: login: franciscojavierarceo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-24T16:42:23Z
user: login: franciscojavierarceo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-24T16:47:35Z
user: login: franciscojavierarceo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-24T16:48:30Z
user: login: franciscojavierarceo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-24T16:48:56Z
user: login: franciscojavierarceo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-24T16:49:48Z
user: login: franciscojavierarceo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-24T16:50:04Z
user: login: franciscojavierarceo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-24T16:50:26Z
user: login: franciscojavierarceo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-24T16:50:39Z
user: login: franciscojavierarceo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-24T16:51:07Z
user: login: franciscojavierarceo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-24T16:52:13Z
user: login: franciscojavierarceo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-24T16:52:24Z
user: login: franciscojavierarceo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-24T16:53:05Z
user: login: franciscojavierarceo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-24T16:53:44Z
user: login: franciscojavierarceo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-24T16:54:38Z
user: login: franciscojavierarceo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-24T16:55:00Z
user: login: franciscojavierarceo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-24T16:55:25Z
user: login: franciscojavierarceo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-24T16:57:11Z
user: login: franciscojavierarceo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-24T16:57:33Z
user: login: franciscojavierarceo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-24T16:57:52Z
user: login: franciscojavierarceo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-24T16:58:30Z
user: login: franciscojavierarceo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-24T16:58:50Z
user: login: franciscojavierarceo
body: This is great!!! I left some feedback 🙏 
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-24T17:08:58Z
user: login: akeed
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-24T19:14:44Z
user: login: akeed
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-24T19:19:17Z
user: login: akeed
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-24T19:42:55Z
user: login: akeed
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-24T19:44:10Z
user: login: akeed
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-24T19:44:13Z
user: login: akeed
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-24T19:44:16Z
user: login: akeed
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-24T19:44:21Z
user: login: franciscojavierarceo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-26T15:40:16Z
user: login: franciscojavierarceo
body: @varodrig can you take a look? 
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-26T15:46:26Z
user: login: akeed
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-26T16:14:16Z
user: login: rimolive
body: Left some comments, but the article looks great!
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-26T17:01:53Z
user: login: akeed
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-26T21:18:55Z
user: login: akeed
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-26T21:25:07Z
user: login: hbelmiro
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-27T11:26:39Z
user: login: varodrig
body: This is a great blog! thank you so much for your contributions.
I enjoyed reading it. I left a couple of suggestions.

Additionally, I think it'll be great to include some videos , it could be a link to the demos https://docs.sdv.dev/sdv/demos or if you already have something it will be a nice way to show the framework.
state: COMMENTED
author_association: NONE
submitted_at: 2025-02-28T03:28:02Z
user: login: tarekabouzeid
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-01T13:02:23Z
user: login: akeed
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-01T15:00:45Z
user: login: akeed
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-01T15:05:33Z
user: login: akeed
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-01T15:09:26Z
user: login: akeed
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-01T15:21:52Z
user: login: akeed
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-01T15:55:21Z
user: login: akeed
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-01T15:55:47Z
user: login: franciscojavierarceo
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-02T15:27:07Z

Review comments:
diff_hunk: @@ -0,0 +1,190 @@
+---
+title: "AI-Generated Synthetic Data, using Kubeflow pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic data - why and how?
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: franciscojavierarceo
body: ```suggestion
## Synthetic Data Generation - Why and How?
```
created_at: 2025-02-24T16:31:59Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,190 @@
+---
+title: "AI-Generated Synthetic Data, using Kubeflow pipelines"
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: franciscojavierarceo
body: ```suggestion
title: "Synthetic Data Generation with Kubeflow Pipelines"
```
created_at: 2025-02-24T16:32:25Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,190 @@
+---
+title: "AI-Generated Synthetic Data, using Kubeflow pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic data - why and how?
+
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: franciscojavierarceo
body: First I'd outline what Synthetic Data Generation (SDG) is, what KFP is, and how they work together at a high level.
created_at: 2025-02-24T16:34:05Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,190 @@
+---
+title: "AI-Generated Synthetic Data, using Kubeflow pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic data - why and how?
+
+The best results come from real data, but accessing it often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete.
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: franciscojavierarceo
body: >The best results come from real data, but accessing it often requires lengthy security and legal processes.
I think it'd be good to outline what we're try ing to do here and providing context on the problem we're trying to solve.
created_at: 2025-02-24T16:35:09Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,190 @@
+---
+title: "AI-Generated Synthetic Data, using Kubeflow pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic data - why and how?
+
+The best results come from real data, but accessing it often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete.
+
+While the above focuses on speed and augmentation, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: franciscojavierarceo
body: I like this sentence but I think it's missing something about what we're trying to improve. The speed of development for building models? 
created_at: 2025-02-24T16:35:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,190 @@
+---
+title: "AI-Generated Synthetic Data, using Kubeflow pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic data - why and how?
+
+The best results come from real data, but accessing it often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete.
+
+While the above focuses on speed and augmentation, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is it, how do we evaluate?
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: franciscojavierarceo
body: ```suggestion
But, how good is synthetic data, how do we evaluate it?
```
created_at: 2025-02-24T16:38:40Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,190 @@
+---
+title: "AI-Generated Synthetic Data, using Kubeflow pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic data - why and how?
+
+The best results come from real data, but accessing it often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete.
+
+While the above focuses on speed and augmentation, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is it, how do we evaluate?
+
+### Evaluation criteria for synthetic data - how do we know it's useful?
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: franciscojavierarceo
body: ```suggestion
### Evaluation Criteria for Synthetic Data
```
created_at: 2025-02-24T16:38:59Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,190 @@
+---
+title: "AI-Generated Synthetic Data, using Kubeflow pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic data - why and how?
+
+The best results come from real data, but accessing it often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete.
+
+While the above focuses on speed and augmentation, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is it, how do we evaluate?
+
+### Evaluation criteria for synthetic data - how do we know it's useful?
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthesizer is best for our specific dataset and use case.
+
+We need to ensure a good balance between: 
+
+  - Usability (how useful is the synthetic data for your use case), 
+  - Fidelity (how similar), and 
+  - Privacy (approved level for use case)
+
+For now, we are focusing only on usability and fidelity, using framework-provided measurements for fidelity and workflows described below for usability.
+
+### Methods and frameworks creating synthetic data 
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: franciscojavierarceo
body: ```suggestion
### Frameworks for Creating Synthetic Data 
```
created_at: 2025-02-24T16:39:51Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,190 @@
+---
+title: "AI-Generated Synthetic Data, using Kubeflow pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic data - why and how?
+
+The best results come from real data, but accessing it often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete.
+
+While the above focuses on speed and augmentation, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is it, how do we evaluate?
+
+### Evaluation criteria for synthetic data - how do we know it's useful?
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthesizer is best for our specific dataset and use case.
+
+We need to ensure a good balance between: 
+
+  - Usability (how useful is the synthetic data for your use case), 
+  - Fidelity (how similar), and 
+  - Privacy (approved level for use case)
+
+For now, we are focusing only on usability and fidelity, using framework-provided measurements for fidelity and workflows described below for usability.
+
+### Methods and frameworks creating synthetic data 
+
+For us, open source frameworks are the only ones of interest. 
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: franciscojavierarceo
body: ```suggestion
This post focuses exclusively on open source frameworks.
```
created_at: 2025-02-24T16:40:14Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,190 @@
+---
+title: "AI-Generated Synthetic Data, using Kubeflow pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic data - why and how?
+
+The best results come from real data, but accessing it often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete.
+
+While the above focuses on speed and augmentation, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is it, how do we evaluate?
+
+### Evaluation criteria for synthetic data - how do we know it's useful?
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthesizer is best for our specific dataset and use case.
+
+We need to ensure a good balance between: 
+
+  - Usability (how useful is the synthetic data for your use case), 
+  - Fidelity (how similar), and 
+  - Privacy (approved level for use case)
+
+For now, we are focusing only on usability and fidelity, using framework-provided measurements for fidelity and workflows described below for usability.
+
+### Methods and frameworks creating synthetic data 
+
+For us, open source frameworks are the only ones of interest. 
+Cloud-based solutions that require sending data samples to the cloud miss the whole point — some of our data cannot be sent to the cloud. 
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: franciscojavierarceo
body: ```suggestion
Some data cannot be sent to the cloud, so some cloud-based SDG solutions are not always a good fit. 
```
created_at: 2025-02-24T16:42:06Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,190 @@
+---
+title: "AI-Generated Synthetic Data, using Kubeflow pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic data - why and how?
+
+The best results come from real data, but accessing it often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete.
+
+While the above focuses on speed and augmentation, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is it, how do we evaluate?
+
+### Evaluation criteria for synthetic data - how do we know it's useful?
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthesizer is best for our specific dataset and use case.
+
+We need to ensure a good balance between: 
+
+  - Usability (how useful is the synthetic data for your use case), 
+  - Fidelity (how similar), and 
+  - Privacy (approved level for use case)
+
+For now, we are focusing only on usability and fidelity, using framework-provided measurements for fidelity and workflows described below for usability.
+
+### Methods and frameworks creating synthetic data 
+
+For us, open source frameworks are the only ones of interest. 
+Cloud-based solutions that require sending data samples to the cloud miss the whole point — some of our data cannot be sent to the cloud. 
+For data already in cloud, we can use other cloud-based frameworks.
+Synthesizers are motivated by multiple factors, but in this context, our focus remains on generating synthetic data for on-premise use.
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: franciscojavierarceo
body: We should define synthesizer first somewhere
created_at: 2025-02-24T16:42:23Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,190 @@
+---
+title: "AI-Generated Synthetic Data, using Kubeflow pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic data - why and how?
+
+The best results come from real data, but accessing it often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete.
+
+While the above focuses on speed and augmentation, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is it, how do we evaluate?
+
+### Evaluation criteria for synthetic data - how do we know it's useful?
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthesizer is best for our specific dataset and use case.
+
+We need to ensure a good balance between: 
+
+  - Usability (how useful is the synthetic data for your use case), 
+  - Fidelity (how similar), and 
+  - Privacy (approved level for use case)
+
+For now, we are focusing only on usability and fidelity, using framework-provided measurements for fidelity and workflows described below for usability.
+
+### Methods and frameworks creating synthetic data 
+
+For us, open source frameworks are the only ones of interest. 
+Cloud-based solutions that require sending data samples to the cloud miss the whole point — some of our data cannot be sent to the cloud. 
+For data already in cloud, we can use other cloud-based frameworks.
+Synthesizers are motivated by multiple factors, but in this context, our focus remains on generating synthetic data for on-premise use.
+
+So, what framework did we (initially) choose? Currently we are using the open source version of [SDV](https://sdv.dev/), 
+an easy-to-use framework with a strong community and many useful features out-of-the-box (e.g. built-in evaluators, many modeling techniques). 
+The field of synthetic data is evolving rapidly. While we do not aim to cover the latest advancements exhaustively, the use of foundational models is certainly an area of interest.
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: franciscojavierarceo
body: ```suggestion
The field of synthetic data is evolving rapidly. While we do not aim to cover the latest advancements exhaustively, the use of Foundation models is certainly an area of interest.
```
created_at: 2025-02-24T16:47:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,190 @@
+---
+title: "AI-Generated Synthetic Data, using Kubeflow pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic data - why and how?
+
+The best results come from real data, but accessing it often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete.
+
+While the above focuses on speed and augmentation, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is it, how do we evaluate?
+
+### Evaluation criteria for synthetic data - how do we know it's useful?
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthesizer is best for our specific dataset and use case.
+
+We need to ensure a good balance between: 
+
+  - Usability (how useful is the synthetic data for your use case), 
+  - Fidelity (how similar), and 
+  - Privacy (approved level for use case)
+
+For now, we are focusing only on usability and fidelity, using framework-provided measurements for fidelity and workflows described below for usability.
+
+### Methods and frameworks creating synthetic data 
+
+For us, open source frameworks are the only ones of interest. 
+Cloud-based solutions that require sending data samples to the cloud miss the whole point — some of our data cannot be sent to the cloud. 
+For data already in cloud, we can use other cloud-based frameworks.
+Synthesizers are motivated by multiple factors, but in this context, our focus remains on generating synthetic data for on-premise use.
+
+So, what framework did we (initially) choose? Currently we are using the open source version of [SDV](https://sdv.dev/), 
+an easy-to-use framework with a strong community and many useful features out-of-the-box (e.g. built-in evaluators, many modeling techniques). 
+The field of synthetic data is evolving rapidly. While we do not aim to cover the latest advancements exhaustively, the use of foundational models is certainly an area of interest.
+
+**Comments on privacy and privacy preserving techniques**
+
+Ensuring privacy in synthetic data is a non-trivial problem, even if there
+are techniques to ensure levels of privacy, it remains an active area of research. 
+
+*Privacy problems, in synthetic data?*
+
+Yes, when creating synthetic data we build models, models from which we generate synthetic data.
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: franciscojavierarceo
body: ```suggestion
When creating synthetic data we build models, models from which we generate synthetic data, which can expose privacy problems.
```

created_at: 2025-02-24T16:48:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,190 @@
+---
+title: "AI-Generated Synthetic Data, using Kubeflow pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic data - why and how?
+
+The best results come from real data, but accessing it often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete.
+
+While the above focuses on speed and augmentation, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is it, how do we evaluate?
+
+### Evaluation criteria for synthetic data - how do we know it's useful?
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthesizer is best for our specific dataset and use case.
+
+We need to ensure a good balance between: 
+
+  - Usability (how useful is the synthetic data for your use case), 
+  - Fidelity (how similar), and 
+  - Privacy (approved level for use case)
+
+For now, we are focusing only on usability and fidelity, using framework-provided measurements for fidelity and workflows described below for usability.
+
+### Methods and frameworks creating synthetic data 
+
+For us, open source frameworks are the only ones of interest. 
+Cloud-based solutions that require sending data samples to the cloud miss the whole point — some of our data cannot be sent to the cloud. 
+For data already in cloud, we can use other cloud-based frameworks.
+Synthesizers are motivated by multiple factors, but in this context, our focus remains on generating synthetic data for on-premise use.
+
+So, what framework did we (initially) choose? Currently we are using the open source version of [SDV](https://sdv.dev/), 
+an easy-to-use framework with a strong community and many useful features out-of-the-box (e.g. built-in evaluators, many modeling techniques). 
+The field of synthetic data is evolving rapidly. While we do not aim to cover the latest advancements exhaustively, the use of foundational models is certainly an area of interest.
+
+**Comments on privacy and privacy preserving techniques**
+
+Ensuring privacy in synthetic data is a non-trivial problem, even if there
+are techniques to ensure levels of privacy, it remains an active area of research. 
+
+*Privacy problems, in synthetic data?*
+
+Yes, when creating synthetic data we build models, models from which we generate synthetic data.
+However, these models can e.g. be overfitted, resulting in leakage of the training data (the real data). 
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: franciscojavierarceo
body: ```suggestion
However, these models can be overfit, resulting in leakage of the training data (i.e., the real data). 
```
created_at: 2025-02-24T16:48:56Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,190 @@
+---
+title: "AI-Generated Synthetic Data, using Kubeflow pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic data - why and how?
+
+The best results come from real data, but accessing it often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete.
+
+While the above focuses on speed and augmentation, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is it, how do we evaluate?
+
+### Evaluation criteria for synthetic data - how do we know it's useful?
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthesizer is best for our specific dataset and use case.
+
+We need to ensure a good balance between: 
+
+  - Usability (how useful is the synthetic data for your use case), 
+  - Fidelity (how similar), and 
+  - Privacy (approved level for use case)
+
+For now, we are focusing only on usability and fidelity, using framework-provided measurements for fidelity and workflows described below for usability.
+
+### Methods and frameworks creating synthetic data 
+
+For us, open source frameworks are the only ones of interest. 
+Cloud-based solutions that require sending data samples to the cloud miss the whole point — some of our data cannot be sent to the cloud. 
+For data already in cloud, we can use other cloud-based frameworks.
+Synthesizers are motivated by multiple factors, but in this context, our focus remains on generating synthetic data for on-premise use.
+
+So, what framework did we (initially) choose? Currently we are using the open source version of [SDV](https://sdv.dev/), 
+an easy-to-use framework with a strong community and many useful features out-of-the-box (e.g. built-in evaluators, many modeling techniques). 
+The field of synthetic data is evolving rapidly. While we do not aim to cover the latest advancements exhaustively, the use of foundational models is certainly an area of interest.
+
+**Comments on privacy and privacy preserving techniques**
+
+Ensuring privacy in synthetic data is a non-trivial problem, even if there
+are techniques to ensure levels of privacy, it remains an active area of research. 
+
+*Privacy problems, in synthetic data?*
+
+Yes, when creating synthetic data we build models, models from which we generate synthetic data.
+However, these models can e.g. be overfitted, resulting in leakage of the training data (the real data). 
+Another challenge lies in handling anomalies. If certain information in the real data stands out and we don't account for it when creating synthetic data, there is a risk of backtracking from the synthetic data to the real.
+A typical example could be 'a very rich person in the dataset' or 'a very rare disease'. 
+The problem here, besides from the privacy part, is that it might be the anomalies we really are looking for.
+
+We are experimenting with various differential privacy strategies, but it is still early days, and we do not focus on them in the examples below.
+
+### The Synthetic Data Vault (SDV) - high level
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: franciscojavierarceo
body: ```suggestion
### The Synthetic Data Vault (SDV)
```
created_at: 2025-02-24T16:49:47Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,190 @@
+---
+title: "AI-Generated Synthetic Data, using Kubeflow pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic data - why and how?
+
+The best results come from real data, but accessing it often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete.
+
+While the above focuses on speed and augmentation, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is it, how do we evaluate?
+
+### Evaluation criteria for synthetic data - how do we know it's useful?
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthesizer is best for our specific dataset and use case.
+
+We need to ensure a good balance between: 
+
+  - Usability (how useful is the synthetic data for your use case), 
+  - Fidelity (how similar), and 
+  - Privacy (approved level for use case)
+
+For now, we are focusing only on usability and fidelity, using framework-provided measurements for fidelity and workflows described below for usability.
+
+### Methods and frameworks creating synthetic data 
+
+For us, open source frameworks are the only ones of interest. 
+Cloud-based solutions that require sending data samples to the cloud miss the whole point — some of our data cannot be sent to the cloud. 
+For data already in cloud, we can use other cloud-based frameworks.
+Synthesizers are motivated by multiple factors, but in this context, our focus remains on generating synthetic data for on-premise use.
+
+So, what framework did we (initially) choose? Currently we are using the open source version of [SDV](https://sdv.dev/), 
+an easy-to-use framework with a strong community and many useful features out-of-the-box (e.g. built-in evaluators, many modeling techniques). 
+The field of synthetic data is evolving rapidly. While we do not aim to cover the latest advancements exhaustively, the use of foundational models is certainly an area of interest.
+
+**Comments on privacy and privacy preserving techniques**
+
+Ensuring privacy in synthetic data is a non-trivial problem, even if there
+are techniques to ensure levels of privacy, it remains an active area of research. 
+
+*Privacy problems, in synthetic data?*
+
+Yes, when creating synthetic data we build models, models from which we generate synthetic data.
+However, these models can e.g. be overfitted, resulting in leakage of the training data (the real data). 
+Another challenge lies in handling anomalies. If certain information in the real data stands out and we don't account for it when creating synthetic data, there is a risk of backtracking from the synthetic data to the real.
+A typical example could be 'a very rich person in the dataset' or 'a very rare disease'. 
+The problem here, besides from the privacy part, is that it might be the anomalies we really are looking for.
+
+We are experimenting with various differential privacy strategies, but it is still early days, and we do not focus on them in the examples below.
+
+### The Synthetic Data Vault (SDV) - high level
+
+When you initialize and fit a synthesizer (like GaussianCopulaSynthesizer, CTGANSynthesizer, etc.), it trains a model based on 
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: franciscojavierarceo
body: Could you link to some of those syntheizer resources?
created_at: 2025-02-24T16:50:04Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,190 @@
+---
+title: "AI-Generated Synthetic Data, using Kubeflow pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic data - why and how?
+
+The best results come from real data, but accessing it often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete.
+
+While the above focuses on speed and augmentation, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is it, how do we evaluate?
+
+### Evaluation criteria for synthetic data - how do we know it's useful?
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthesizer is best for our specific dataset and use case.
+
+We need to ensure a good balance between: 
+
+  - Usability (how useful is the synthetic data for your use case), 
+  - Fidelity (how similar), and 
+  - Privacy (approved level for use case)
+
+For now, we are focusing only on usability and fidelity, using framework-provided measurements for fidelity and workflows described below for usability.
+
+### Methods and frameworks creating synthetic data 
+
+For us, open source frameworks are the only ones of interest. 
+Cloud-based solutions that require sending data samples to the cloud miss the whole point — some of our data cannot be sent to the cloud. 
+For data already in cloud, we can use other cloud-based frameworks.
+Synthesizers are motivated by multiple factors, but in this context, our focus remains on generating synthetic data for on-premise use.
+
+So, what framework did we (initially) choose? Currently we are using the open source version of [SDV](https://sdv.dev/), 
+an easy-to-use framework with a strong community and many useful features out-of-the-box (e.g. built-in evaluators, many modeling techniques). 
+The field of synthetic data is evolving rapidly. While we do not aim to cover the latest advancements exhaustively, the use of foundational models is certainly an area of interest.
+
+**Comments on privacy and privacy preserving techniques**
+
+Ensuring privacy in synthetic data is a non-trivial problem, even if there
+are techniques to ensure levels of privacy, it remains an active area of research. 
+
+*Privacy problems, in synthetic data?*
+
+Yes, when creating synthetic data we build models, models from which we generate synthetic data.
+However, these models can e.g. be overfitted, resulting in leakage of the training data (the real data). 
+Another challenge lies in handling anomalies. If certain information in the real data stands out and we don't account for it when creating synthetic data, there is a risk of backtracking from the synthetic data to the real.
+A typical example could be 'a very rich person in the dataset' or 'a very rare disease'. 
+The problem here, besides from the privacy part, is that it might be the anomalies we really are looking for.
+
+We are experimenting with various differential privacy strategies, but it is still early days, and we do not focus on them in the examples below.
+
+### The Synthetic Data Vault (SDV) - high level
+
+When you initialize and fit a synthesizer (like GaussianCopulaSynthesizer, CTGANSynthesizer, etc.), it trains a model based on 
+the dataset you provide. This model learns the distribution of the data, capturing the relationships and dependencies between 
+different features in the dataset. Below are the (free) synthesizers provided by SDV that we evaluated on each use case.
+
+The synthesizer doesn't memorize individual records from the dataset. 
+Instead, it tries to learn the underlying statistical patterns, correlations, and distributions present in the data. 
+Each synthesizer does this differently:
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: franciscojavierarceo
body: nice
created_at: 2025-02-24T16:50:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,190 @@
+---
+title: "AI-Generated Synthetic Data, using Kubeflow pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic data - why and how?
+
+The best results come from real data, but accessing it often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete.
+
+While the above focuses on speed and augmentation, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is it, how do we evaluate?
+
+### Evaluation criteria for synthetic data - how do we know it's useful?
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthesizer is best for our specific dataset and use case.
+
+We need to ensure a good balance between: 
+
+  - Usability (how useful is the synthetic data for your use case), 
+  - Fidelity (how similar), and 
+  - Privacy (approved level for use case)
+
+For now, we are focusing only on usability and fidelity, using framework-provided measurements for fidelity and workflows described below for usability.
+
+### Methods and frameworks creating synthetic data 
+
+For us, open source frameworks are the only ones of interest. 
+Cloud-based solutions that require sending data samples to the cloud miss the whole point — some of our data cannot be sent to the cloud. 
+For data already in cloud, we can use other cloud-based frameworks.
+Synthesizers are motivated by multiple factors, but in this context, our focus remains on generating synthetic data for on-premise use.
+
+So, what framework did we (initially) choose? Currently we are using the open source version of [SDV](https://sdv.dev/), 
+an easy-to-use framework with a strong community and many useful features out-of-the-box (e.g. built-in evaluators, many modeling techniques). 
+The field of synthetic data is evolving rapidly. While we do not aim to cover the latest advancements exhaustively, the use of foundational models is certainly an area of interest.
+
+**Comments on privacy and privacy preserving techniques**
+
+Ensuring privacy in synthetic data is a non-trivial problem, even if there
+are techniques to ensure levels of privacy, it remains an active area of research. 
+
+*Privacy problems, in synthetic data?*
+
+Yes, when creating synthetic data we build models, models from which we generate synthetic data.
+However, these models can e.g. be overfitted, resulting in leakage of the training data (the real data). 
+Another challenge lies in handling anomalies. If certain information in the real data stands out and we don't account for it when creating synthetic data, there is a risk of backtracking from the synthetic data to the real.
+A typical example could be 'a very rich person in the dataset' or 'a very rare disease'. 
+The problem here, besides from the privacy part, is that it might be the anomalies we really are looking for.
+
+We are experimenting with various differential privacy strategies, but it is still early days, and we do not focus on them in the examples below.
+
+### The Synthetic Data Vault (SDV) - high level
+
+When you initialize and fit a synthesizer (like GaussianCopulaSynthesizer, CTGANSynthesizer, etc.), it trains a model based on 
+the dataset you provide. This model learns the distribution of the data, capturing the relationships and dependencies between 
+different features in the dataset. Below are the (free) synthesizers provided by SDV that we evaluated on each use case.
+
+The synthesizer doesn't memorize individual records from the dataset. 
+Instead, it tries to learn the underlying statistical patterns, correlations, and distributions present in the data. 
+Each synthesizer does this differently:
+
+- **GaussianCopulaSynthesizer:** Models the distribution of each feature using copulas and marginal distributions. 
+- **CTGANSynthesizer:** Uses a GAN (Generative Adversarial Network) approach, which involves training two neural networks (a generator and a discriminator) to model complex data distributions, especially useful for discrete or mixed-type data.
+- **TVAESynthesizer:** Uses a Variational Autoencoder (VAE) approach to learn a lower-dimensional representation of the data and then generate new samples.
+- **CopulaGANSynthesizer:** Combines elements of copulas and GANs to capture complex dependencies.
+- **PARSynthesizer:** Uses Probabilistic AutoRegressive models to model sequential dependencies in the data.
+
+*There are more synthesizers, also from SDV, but not all open source.* We used the first four, when evaluating optimal synthesizer for our different use cases.
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: franciscojavierarceo
body: ```suggestion
*There are more synthesizers, also from SDV, but not all are open source.* We used the first four, when evaluating optimal synthesizer for our different use cases.
```
created_at: 2025-02-24T16:50:39Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,190 @@
+---
+title: "AI-Generated Synthetic Data, using Kubeflow pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic data - why and how?
+
+The best results come from real data, but accessing it often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete.
+
+While the above focuses on speed and augmentation, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is it, how do we evaluate?
+
+### Evaluation criteria for synthetic data - how do we know it's useful?
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthesizer is best for our specific dataset and use case.
+
+We need to ensure a good balance between: 
+
+  - Usability (how useful is the synthetic data for your use case), 
+  - Fidelity (how similar), and 
+  - Privacy (approved level for use case)
+
+For now, we are focusing only on usability and fidelity, using framework-provided measurements for fidelity and workflows described below for usability.
+
+### Methods and frameworks creating synthetic data 
+
+For us, open source frameworks are the only ones of interest. 
+Cloud-based solutions that require sending data samples to the cloud miss the whole point — some of our data cannot be sent to the cloud. 
+For data already in cloud, we can use other cloud-based frameworks.
+Synthesizers are motivated by multiple factors, but in this context, our focus remains on generating synthetic data for on-premise use.
+
+So, what framework did we (initially) choose? Currently we are using the open source version of [SDV](https://sdv.dev/), 
+an easy-to-use framework with a strong community and many useful features out-of-the-box (e.g. built-in evaluators, many modeling techniques). 
+The field of synthetic data is evolving rapidly. While we do not aim to cover the latest advancements exhaustively, the use of foundational models is certainly an area of interest.
+
+**Comments on privacy and privacy preserving techniques**
+
+Ensuring privacy in synthetic data is a non-trivial problem, even if there
+are techniques to ensure levels of privacy, it remains an active area of research. 
+
+*Privacy problems, in synthetic data?*
+
+Yes, when creating synthetic data we build models, models from which we generate synthetic data.
+However, these models can e.g. be overfitted, resulting in leakage of the training data (the real data). 
+Another challenge lies in handling anomalies. If certain information in the real data stands out and we don't account for it when creating synthetic data, there is a risk of backtracking from the synthetic data to the real.
+A typical example could be 'a very rich person in the dataset' or 'a very rare disease'. 
+The problem here, besides from the privacy part, is that it might be the anomalies we really are looking for.
+
+We are experimenting with various differential privacy strategies, but it is still early days, and we do not focus on them in the examples below.
+
+### The Synthetic Data Vault (SDV) - high level
+
+When you initialize and fit a synthesizer (like GaussianCopulaSynthesizer, CTGANSynthesizer, etc.), it trains a model based on 
+the dataset you provide. This model learns the distribution of the data, capturing the relationships and dependencies between 
+different features in the dataset. Below are the (free) synthesizers provided by SDV that we evaluated on each use case.
+
+The synthesizer doesn't memorize individual records from the dataset. 
+Instead, it tries to learn the underlying statistical patterns, correlations, and distributions present in the data. 
+Each synthesizer does this differently:
+
+- **GaussianCopulaSynthesizer:** Models the distribution of each feature using copulas and marginal distributions. 
+- **CTGANSynthesizer:** Uses a GAN (Generative Adversarial Network) approach, which involves training two neural networks (a generator and a discriminator) to model complex data distributions, especially useful for discrete or mixed-type data.
+- **TVAESynthesizer:** Uses a Variational Autoencoder (VAE) approach to learn a lower-dimensional representation of the data and then generate new samples.
+- **CopulaGANSynthesizer:** Combines elements of copulas and GANs to capture complex dependencies.
+- **PARSynthesizer:** Uses Probabilistic AutoRegressive models to model sequential dependencies in the data.
+
+*There are more synthesizers, also from SDV, but not all open source.* We used the first four, when evaluating optimal synthesizer for our different use cases.
+
+**Generators - generating new data - on demand**
+
+Once trained, the synthesizer uses the learned model to generate new synthetic data that follows the same statistical properties and 
+distributions as the original dataset, without directly copying any real data points. If you need more data? Just call the generator.
+
+### Our on-premise analytics platform - ARCUS
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: franciscojavierarceo
body: ```suggestion
### Our On-Premise Analytics Platform: ARCUS
```
created_at: 2025-02-24T16:51:07Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,190 @@
+---
+title: "AI-Generated Synthetic Data, using Kubeflow pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic data - why and how?
+
+The best results come from real data, but accessing it often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete.
+
+While the above focuses on speed and augmentation, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is it, how do we evaluate?
+
+### Evaluation criteria for synthetic data - how do we know it's useful?
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthesizer is best for our specific dataset and use case.
+
+We need to ensure a good balance between: 
+
+  - Usability (how useful is the synthetic data for your use case), 
+  - Fidelity (how similar), and 
+  - Privacy (approved level for use case)
+
+For now, we are focusing only on usability and fidelity, using framework-provided measurements for fidelity and workflows described below for usability.
+
+### Methods and frameworks creating synthetic data 
+
+For us, open source frameworks are the only ones of interest. 
+Cloud-based solutions that require sending data samples to the cloud miss the whole point — some of our data cannot be sent to the cloud. 
+For data already in cloud, we can use other cloud-based frameworks.
+Synthesizers are motivated by multiple factors, but in this context, our focus remains on generating synthetic data for on-premise use.
+
+So, what framework did we (initially) choose? Currently we are using the open source version of [SDV](https://sdv.dev/), 
+an easy-to-use framework with a strong community and many useful features out-of-the-box (e.g. built-in evaluators, many modeling techniques). 
+The field of synthetic data is evolving rapidly. While we do not aim to cover the latest advancements exhaustively, the use of foundational models is certainly an area of interest.
+
+**Comments on privacy and privacy preserving techniques**
+
+Ensuring privacy in synthetic data is a non-trivial problem, even if there
+are techniques to ensure levels of privacy, it remains an active area of research. 
+
+*Privacy problems, in synthetic data?*
+
+Yes, when creating synthetic data we build models, models from which we generate synthetic data.
+However, these models can e.g. be overfitted, resulting in leakage of the training data (the real data). 
+Another challenge lies in handling anomalies. If certain information in the real data stands out and we don't account for it when creating synthetic data, there is a risk of backtracking from the synthetic data to the real.
+A typical example could be 'a very rich person in the dataset' or 'a very rare disease'. 
+The problem here, besides from the privacy part, is that it might be the anomalies we really are looking for.
+
+We are experimenting with various differential privacy strategies, but it is still early days, and we do not focus on them in the examples below.
+
+### The Synthetic Data Vault (SDV) - high level
+
+When you initialize and fit a synthesizer (like GaussianCopulaSynthesizer, CTGANSynthesizer, etc.), it trains a model based on 
+the dataset you provide. This model learns the distribution of the data, capturing the relationships and dependencies between 
+different features in the dataset. Below are the (free) synthesizers provided by SDV that we evaluated on each use case.
+
+The synthesizer doesn't memorize individual records from the dataset. 
+Instead, it tries to learn the underlying statistical patterns, correlations, and distributions present in the data. 
+Each synthesizer does this differently:
+
+- **GaussianCopulaSynthesizer:** Models the distribution of each feature using copulas and marginal distributions. 
+- **CTGANSynthesizer:** Uses a GAN (Generative Adversarial Network) approach, which involves training two neural networks (a generator and a discriminator) to model complex data distributions, especially useful for discrete or mixed-type data.
+- **TVAESynthesizer:** Uses a Variational Autoencoder (VAE) approach to learn a lower-dimensional representation of the data and then generate new samples.
+- **CopulaGANSynthesizer:** Combines elements of copulas and GANs to capture complex dependencies.
+- **PARSynthesizer:** Uses Probabilistic AutoRegressive models to model sequential dependencies in the data.
+
+*There are more synthesizers, also from SDV, but not all open source.* We used the first four, when evaluating optimal synthesizer for our different use cases.
+
+**Generators - generating new data - on demand**
+
+Once trained, the synthesizer uses the learned model to generate new synthetic data that follows the same statistical properties and 
+distributions as the original dataset, without directly copying any real data points. If you need more data? Just call the generator.
+
+### Our on-premise analytics platform - ARCUS
+
+<div style="display: flex; align-items: center; gap: 20px;">
+  <p>
+    ARCUS is Telia’s advanced on-premise analytics platform, designed to support a wide range of use cases.
+    The platform provides a Kubeflow-based MLOps environment for descriptive, predictive, generative, and (ongoing) agentic AI. 
+    Fully built on open-source, ARCUS integrates a comprehensive stack of components into a unified platform - where Kubernetes is the cornerstone.
+  </p>
+</div>
+
+### Needed environment to create synthetic data
+
+For an efficient, automated selection of the best synthesizer, we need a number of things - from the underlying platform with GPUs and MLOps (Kubeflow).
+
+- Kubeflow pipelines
+- GPU capabilities (for performance and efficiency)
+- Development (IDE) environment (for framework building and running)
+- Modern data platform (MinIO, Airflow) automating the synthetic data generation datasets
+
+#### Parallelism needed
+
+In the (Kube)flows below, we run evaluations in parallel - one for respective synthesizer, followed by a comparison of usability and fidelity scores, selecting the 'winner'.
+
+*Note:* In earlier version of Kubeflow we noticed that the parallelism wasn't acting as expected, waiting for all threads to complete before moving to next step. We had to create a temporary workaround for this, now solved in Kubeflow Pipelines 2.3.0.
+
+Below, we briefly describe the base flow for selecting synthesizer, followed by one use case where we use the resulting data generator for ML development in cloud.
+
+## Exploring the creation and usefulness of synthetic data
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: franciscojavierarceo
body: ```suggestion
## Exploring the Creation and Usefulness of Synthetic Data
```
created_at: 2025-02-24T16:52:12Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,190 @@
+---
+title: "AI-Generated Synthetic Data, using Kubeflow pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic data - why and how?
+
+The best results come from real data, but accessing it often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete.
+
+While the above focuses on speed and augmentation, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is it, how do we evaluate?
+
+### Evaluation criteria for synthetic data - how do we know it's useful?
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthesizer is best for our specific dataset and use case.
+
+We need to ensure a good balance between: 
+
+  - Usability (how useful is the synthetic data for your use case), 
+  - Fidelity (how similar), and 
+  - Privacy (approved level for use case)
+
+For now, we are focusing only on usability and fidelity, using framework-provided measurements for fidelity and workflows described below for usability.
+
+### Methods and frameworks creating synthetic data 
+
+For us, open source frameworks are the only ones of interest. 
+Cloud-based solutions that require sending data samples to the cloud miss the whole point — some of our data cannot be sent to the cloud. 
+For data already in cloud, we can use other cloud-based frameworks.
+Synthesizers are motivated by multiple factors, but in this context, our focus remains on generating synthetic data for on-premise use.
+
+So, what framework did we (initially) choose? Currently we are using the open source version of [SDV](https://sdv.dev/), 
+an easy-to-use framework with a strong community and many useful features out-of-the-box (e.g. built-in evaluators, many modeling techniques). 
+The field of synthetic data is evolving rapidly. While we do not aim to cover the latest advancements exhaustively, the use of foundational models is certainly an area of interest.
+
+**Comments on privacy and privacy preserving techniques**
+
+Ensuring privacy in synthetic data is a non-trivial problem, even if there
+are techniques to ensure levels of privacy, it remains an active area of research. 
+
+*Privacy problems, in synthetic data?*
+
+Yes, when creating synthetic data we build models, models from which we generate synthetic data.
+However, these models can e.g. be overfitted, resulting in leakage of the training data (the real data). 
+Another challenge lies in handling anomalies. If certain information in the real data stands out and we don't account for it when creating synthetic data, there is a risk of backtracking from the synthetic data to the real.
+A typical example could be 'a very rich person in the dataset' or 'a very rare disease'. 
+The problem here, besides from the privacy part, is that it might be the anomalies we really are looking for.
+
+We are experimenting with various differential privacy strategies, but it is still early days, and we do not focus on them in the examples below.
+
+### The Synthetic Data Vault (SDV) - high level
+
+When you initialize and fit a synthesizer (like GaussianCopulaSynthesizer, CTGANSynthesizer, etc.), it trains a model based on 
+the dataset you provide. This model learns the distribution of the data, capturing the relationships and dependencies between 
+different features in the dataset. Below are the (free) synthesizers provided by SDV that we evaluated on each use case.
+
+The synthesizer doesn't memorize individual records from the dataset. 
+Instead, it tries to learn the underlying statistical patterns, correlations, and distributions present in the data. 
+Each synthesizer does this differently:
+
+- **GaussianCopulaSynthesizer:** Models the distribution of each feature using copulas and marginal distributions. 
+- **CTGANSynthesizer:** Uses a GAN (Generative Adversarial Network) approach, which involves training two neural networks (a generator and a discriminator) to model complex data distributions, especially useful for discrete or mixed-type data.
+- **TVAESynthesizer:** Uses a Variational Autoencoder (VAE) approach to learn a lower-dimensional representation of the data and then generate new samples.
+- **CopulaGANSynthesizer:** Combines elements of copulas and GANs to capture complex dependencies.
+- **PARSynthesizer:** Uses Probabilistic AutoRegressive models to model sequential dependencies in the data.
+
+*There are more synthesizers, also from SDV, but not all open source.* We used the first four, when evaluating optimal synthesizer for our different use cases.
+
+**Generators - generating new data - on demand**
+
+Once trained, the synthesizer uses the learned model to generate new synthetic data that follows the same statistical properties and 
+distributions as the original dataset, without directly copying any real data points. If you need more data? Just call the generator.
+
+### Our on-premise analytics platform - ARCUS
+
+<div style="display: flex; align-items: center; gap: 20px;">
+  <p>
+    ARCUS is Telia’s advanced on-premise analytics platform, designed to support a wide range of use cases.
+    The platform provides a Kubeflow-based MLOps environment for descriptive, predictive, generative, and (ongoing) agentic AI. 
+    Fully built on open-source, ARCUS integrates a comprehensive stack of components into a unified platform - where Kubernetes is the cornerstone.
+  </p>
+</div>
+
+### Needed environment to create synthetic data
+
+For an efficient, automated selection of the best synthesizer, we need a number of things - from the underlying platform with GPUs and MLOps (Kubeflow).
+
+- Kubeflow pipelines
+- GPU capabilities (for performance and efficiency)
+- Development (IDE) environment (for framework building and running)
+- Modern data platform (MinIO, Airflow) automating the synthetic data generation datasets
+
+#### Parallelism needed
+
+In the (Kube)flows below, we run evaluations in parallel - one for respective synthesizer, followed by a comparison of usability and fidelity scores, selecting the 'winner'.
+
+*Note:* In earlier version of Kubeflow we noticed that the parallelism wasn't acting as expected, waiting for all threads to complete before moving to next step. We had to create a temporary workaround for this, now solved in Kubeflow Pipelines 2.3.0.
+
+Below, we briefly describe the base flow for selecting synthesizer, followed by one use case where we use the resulting data generator for ML development in cloud.
+
+## Exploring the creation and usefulness of synthetic data
+
+This is what we want to do: We have a use case, the supporting data, and developed ML model.
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: franciscojavierarceo
body: ```suggestion
This is what we want to do: we have a use case, the supporting data, and developed ML model.
```
created_at: 2025-02-24T16:52:24Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,190 @@
+---
+title: "AI-Generated Synthetic Data, using Kubeflow pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic data - why and how?
+
+The best results come from real data, but accessing it often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete.
+
+While the above focuses on speed and augmentation, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is it, how do we evaluate?
+
+### Evaluation criteria for synthetic data - how do we know it's useful?
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthesizer is best for our specific dataset and use case.
+
+We need to ensure a good balance between: 
+
+  - Usability (how useful is the synthetic data for your use case), 
+  - Fidelity (how similar), and 
+  - Privacy (approved level for use case)
+
+For now, we are focusing only on usability and fidelity, using framework-provided measurements for fidelity and workflows described below for usability.
+
+### Methods and frameworks creating synthetic data 
+
+For us, open source frameworks are the only ones of interest. 
+Cloud-based solutions that require sending data samples to the cloud miss the whole point — some of our data cannot be sent to the cloud. 
+For data already in cloud, we can use other cloud-based frameworks.
+Synthesizers are motivated by multiple factors, but in this context, our focus remains on generating synthetic data for on-premise use.
+
+So, what framework did we (initially) choose? Currently we are using the open source version of [SDV](https://sdv.dev/), 
+an easy-to-use framework with a strong community and many useful features out-of-the-box (e.g. built-in evaluators, many modeling techniques). 
+The field of synthetic data is evolving rapidly. While we do not aim to cover the latest advancements exhaustively, the use of foundational models is certainly an area of interest.
+
+**Comments on privacy and privacy preserving techniques**
+
+Ensuring privacy in synthetic data is a non-trivial problem, even if there
+are techniques to ensure levels of privacy, it remains an active area of research. 
+
+*Privacy problems, in synthetic data?*
+
+Yes, when creating synthetic data we build models, models from which we generate synthetic data.
+However, these models can e.g. be overfitted, resulting in leakage of the training data (the real data). 
+Another challenge lies in handling anomalies. If certain information in the real data stands out and we don't account for it when creating synthetic data, there is a risk of backtracking from the synthetic data to the real.
+A typical example could be 'a very rich person in the dataset' or 'a very rare disease'. 
+The problem here, besides from the privacy part, is that it might be the anomalies we really are looking for.
+
+We are experimenting with various differential privacy strategies, but it is still early days, and we do not focus on them in the examples below.
+
+### The Synthetic Data Vault (SDV) - high level
+
+When you initialize and fit a synthesizer (like GaussianCopulaSynthesizer, CTGANSynthesizer, etc.), it trains a model based on 
+the dataset you provide. This model learns the distribution of the data, capturing the relationships and dependencies between 
+different features in the dataset. Below are the (free) synthesizers provided by SDV that we evaluated on each use case.
+
+The synthesizer doesn't memorize individual records from the dataset. 
+Instead, it tries to learn the underlying statistical patterns, correlations, and distributions present in the data. 
+Each synthesizer does this differently:
+
+- **GaussianCopulaSynthesizer:** Models the distribution of each feature using copulas and marginal distributions. 
+- **CTGANSynthesizer:** Uses a GAN (Generative Adversarial Network) approach, which involves training two neural networks (a generator and a discriminator) to model complex data distributions, especially useful for discrete or mixed-type data.
+- **TVAESynthesizer:** Uses a Variational Autoencoder (VAE) approach to learn a lower-dimensional representation of the data and then generate new samples.
+- **CopulaGANSynthesizer:** Combines elements of copulas and GANs to capture complex dependencies.
+- **PARSynthesizer:** Uses Probabilistic AutoRegressive models to model sequential dependencies in the data.
+
+*There are more synthesizers, also from SDV, but not all open source.* We used the first four, when evaluating optimal synthesizer for our different use cases.
+
+**Generators - generating new data - on demand**
+
+Once trained, the synthesizer uses the learned model to generate new synthetic data that follows the same statistical properties and 
+distributions as the original dataset, without directly copying any real data points. If you need more data? Just call the generator.
+
+### Our on-premise analytics platform - ARCUS
+
+<div style="display: flex; align-items: center; gap: 20px;">
+  <p>
+    ARCUS is Telia’s advanced on-premise analytics platform, designed to support a wide range of use cases.
+    The platform provides a Kubeflow-based MLOps environment for descriptive, predictive, generative, and (ongoing) agentic AI. 
+    Fully built on open-source, ARCUS integrates a comprehensive stack of components into a unified platform - where Kubernetes is the cornerstone.
+  </p>
+</div>
+
+### Needed environment to create synthetic data
+
+For an efficient, automated selection of the best synthesizer, we need a number of things - from the underlying platform with GPUs and MLOps (Kubeflow).
+
+- Kubeflow pipelines
+- GPU capabilities (for performance and efficiency)
+- Development (IDE) environment (for framework building and running)
+- Modern data platform (MinIO, Airflow) automating the synthetic data generation datasets
+
+#### Parallelism needed
+
+In the (Kube)flows below, we run evaluations in parallel - one for respective synthesizer, followed by a comparison of usability and fidelity scores, selecting the 'winner'.
+
+*Note:* In earlier version of Kubeflow we noticed that the parallelism wasn't acting as expected, waiting for all threads to complete before moving to next step. We had to create a temporary workaround for this, now solved in Kubeflow Pipelines 2.3.0.
+
+Below, we briefly describe the base flow for selecting synthesizer, followed by one use case where we use the resulting data generator for ML development in cloud.
+
+## Exploring the creation and usefulness of synthetic data
+
+This is what we want to do: We have a use case, the supporting data, and developed ML model.
+
+- How similar is the synthetic data compared to the real data (interesting for e.g. visualization use cases)?
+- How well do the ML models based on synthetic data keep up with ML models based on real data?
+
+**General validation of synthetic data technique**
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: franciscojavierarceo
body: ```suggestion
**Validation of synthetic data techniques**
```
created_at: 2025-02-24T16:53:05Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,190 @@
+---
+title: "AI-Generated Synthetic Data, using Kubeflow pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic data - why and how?
+
+The best results come from real data, but accessing it often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete.
+
+While the above focuses on speed and augmentation, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is it, how do we evaluate?
+
+### Evaluation criteria for synthetic data - how do we know it's useful?
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthesizer is best for our specific dataset and use case.
+
+We need to ensure a good balance between: 
+
+  - Usability (how useful is the synthetic data for your use case), 
+  - Fidelity (how similar), and 
+  - Privacy (approved level for use case)
+
+For now, we are focusing only on usability and fidelity, using framework-provided measurements for fidelity and workflows described below for usability.
+
+### Methods and frameworks creating synthetic data 
+
+For us, open source frameworks are the only ones of interest. 
+Cloud-based solutions that require sending data samples to the cloud miss the whole point — some of our data cannot be sent to the cloud. 
+For data already in cloud, we can use other cloud-based frameworks.
+Synthesizers are motivated by multiple factors, but in this context, our focus remains on generating synthetic data for on-premise use.
+
+So, what framework did we (initially) choose? Currently we are using the open source version of [SDV](https://sdv.dev/), 
+an easy-to-use framework with a strong community and many useful features out-of-the-box (e.g. built-in evaluators, many modeling techniques). 
+The field of synthetic data is evolving rapidly. While we do not aim to cover the latest advancements exhaustively, the use of foundational models is certainly an area of interest.
+
+**Comments on privacy and privacy preserving techniques**
+
+Ensuring privacy in synthetic data is a non-trivial problem, even if there
+are techniques to ensure levels of privacy, it remains an active area of research. 
+
+*Privacy problems, in synthetic data?*
+
+Yes, when creating synthetic data we build models, models from which we generate synthetic data.
+However, these models can e.g. be overfitted, resulting in leakage of the training data (the real data). 
+Another challenge lies in handling anomalies. If certain information in the real data stands out and we don't account for it when creating synthetic data, there is a risk of backtracking from the synthetic data to the real.
+A typical example could be 'a very rich person in the dataset' or 'a very rare disease'. 
+The problem here, besides from the privacy part, is that it might be the anomalies we really are looking for.
+
+We are experimenting with various differential privacy strategies, but it is still early days, and we do not focus on them in the examples below.
+
+### The Synthetic Data Vault (SDV) - high level
+
+When you initialize and fit a synthesizer (like GaussianCopulaSynthesizer, CTGANSynthesizer, etc.), it trains a model based on 
+the dataset you provide. This model learns the distribution of the data, capturing the relationships and dependencies between 
+different features in the dataset. Below are the (free) synthesizers provided by SDV that we evaluated on each use case.
+
+The synthesizer doesn't memorize individual records from the dataset. 
+Instead, it tries to learn the underlying statistical patterns, correlations, and distributions present in the data. 
+Each synthesizer does this differently:
+
+- **GaussianCopulaSynthesizer:** Models the distribution of each feature using copulas and marginal distributions. 
+- **CTGANSynthesizer:** Uses a GAN (Generative Adversarial Network) approach, which involves training two neural networks (a generator and a discriminator) to model complex data distributions, especially useful for discrete or mixed-type data.
+- **TVAESynthesizer:** Uses a Variational Autoencoder (VAE) approach to learn a lower-dimensional representation of the data and then generate new samples.
+- **CopulaGANSynthesizer:** Combines elements of copulas and GANs to capture complex dependencies.
+- **PARSynthesizer:** Uses Probabilistic AutoRegressive models to model sequential dependencies in the data.
+
+*There are more synthesizers, also from SDV, but not all open source.* We used the first four, when evaluating optimal synthesizer for our different use cases.
+
+**Generators - generating new data - on demand**
+
+Once trained, the synthesizer uses the learned model to generate new synthetic data that follows the same statistical properties and 
+distributions as the original dataset, without directly copying any real data points. If you need more data? Just call the generator.
+
+### Our on-premise analytics platform - ARCUS
+
+<div style="display: flex; align-items: center; gap: 20px;">
+  <p>
+    ARCUS is Telia’s advanced on-premise analytics platform, designed to support a wide range of use cases.
+    The platform provides a Kubeflow-based MLOps environment for descriptive, predictive, generative, and (ongoing) agentic AI. 
+    Fully built on open-source, ARCUS integrates a comprehensive stack of components into a unified platform - where Kubernetes is the cornerstone.
+  </p>
+</div>
+
+### Needed environment to create synthetic data
+
+For an efficient, automated selection of the best synthesizer, we need a number of things - from the underlying platform with GPUs and MLOps (Kubeflow).
+
+- Kubeflow pipelines
+- GPU capabilities (for performance and efficiency)
+- Development (IDE) environment (for framework building and running)
+- Modern data platform (MinIO, Airflow) automating the synthetic data generation datasets
+
+#### Parallelism needed
+
+In the (Kube)flows below, we run evaluations in parallel - one for respective synthesizer, followed by a comparison of usability and fidelity scores, selecting the 'winner'.
+
+*Note:* In earlier version of Kubeflow we noticed that the parallelism wasn't acting as expected, waiting for all threads to complete before moving to next step. We had to create a temporary workaround for this, now solved in Kubeflow Pipelines 2.3.0.
+
+Below, we briefly describe the base flow for selecting synthesizer, followed by one use case where we use the resulting data generator for ML development in cloud.
+
+## Exploring the creation and usefulness of synthetic data
+
+This is what we want to do: We have a use case, the supporting data, and developed ML model.
+
+- How similar is the synthetic data compared to the real data (interesting for e.g. visualization use cases)?
+- How well do the ML models based on synthetic data keep up with ML models based on real data?
+
+**General validation of synthetic data technique**
+
+1. Create synthetic data, save the best synthetic data generator. In this step similarity measures are created by the used (SDV) framework 'out of the box'
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: franciscojavierarceo
body: ```suggestion
1. Create the synthetic data and save the best synthetic data generator. In this step similarity measures are created by the out of the box SDV framework
```
created_at: 2025-02-24T16:53:44Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,190 @@
+---
+title: "AI-Generated Synthetic Data, using Kubeflow pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic data - why and how?
+
+The best results come from real data, but accessing it often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete.
+
+While the above focuses on speed and augmentation, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is it, how do we evaluate?
+
+### Evaluation criteria for synthetic data - how do we know it's useful?
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthesizer is best for our specific dataset and use case.
+
+We need to ensure a good balance between: 
+
+  - Usability (how useful is the synthetic data for your use case), 
+  - Fidelity (how similar), and 
+  - Privacy (approved level for use case)
+
+For now, we are focusing only on usability and fidelity, using framework-provided measurements for fidelity and workflows described below for usability.
+
+### Methods and frameworks creating synthetic data 
+
+For us, open source frameworks are the only ones of interest. 
+Cloud-based solutions that require sending data samples to the cloud miss the whole point — some of our data cannot be sent to the cloud. 
+For data already in cloud, we can use other cloud-based frameworks.
+Synthesizers are motivated by multiple factors, but in this context, our focus remains on generating synthetic data for on-premise use.
+
+So, what framework did we (initially) choose? Currently we are using the open source version of [SDV](https://sdv.dev/), 
+an easy-to-use framework with a strong community and many useful features out-of-the-box (e.g. built-in evaluators, many modeling techniques). 
+The field of synthetic data is evolving rapidly. While we do not aim to cover the latest advancements exhaustively, the use of foundational models is certainly an area of interest.
+
+**Comments on privacy and privacy preserving techniques**
+
+Ensuring privacy in synthetic data is a non-trivial problem, even if there
+are techniques to ensure levels of privacy, it remains an active area of research. 
+
+*Privacy problems, in synthetic data?*
+
+Yes, when creating synthetic data we build models, models from which we generate synthetic data.
+However, these models can e.g. be overfitted, resulting in leakage of the training data (the real data). 
+Another challenge lies in handling anomalies. If certain information in the real data stands out and we don't account for it when creating synthetic data, there is a risk of backtracking from the synthetic data to the real.
+A typical example could be 'a very rich person in the dataset' or 'a very rare disease'. 
+The problem here, besides from the privacy part, is that it might be the anomalies we really are looking for.
+
+We are experimenting with various differential privacy strategies, but it is still early days, and we do not focus on them in the examples below.
+
+### The Synthetic Data Vault (SDV) - high level
+
+When you initialize and fit a synthesizer (like GaussianCopulaSynthesizer, CTGANSynthesizer, etc.), it trains a model based on 
+the dataset you provide. This model learns the distribution of the data, capturing the relationships and dependencies between 
+different features in the dataset. Below are the (free) synthesizers provided by SDV that we evaluated on each use case.
+
+The synthesizer doesn't memorize individual records from the dataset. 
+Instead, it tries to learn the underlying statistical patterns, correlations, and distributions present in the data. 
+Each synthesizer does this differently:
+
+- **GaussianCopulaSynthesizer:** Models the distribution of each feature using copulas and marginal distributions. 
+- **CTGANSynthesizer:** Uses a GAN (Generative Adversarial Network) approach, which involves training two neural networks (a generator and a discriminator) to model complex data distributions, especially useful for discrete or mixed-type data.
+- **TVAESynthesizer:** Uses a Variational Autoencoder (VAE) approach to learn a lower-dimensional representation of the data and then generate new samples.
+- **CopulaGANSynthesizer:** Combines elements of copulas and GANs to capture complex dependencies.
+- **PARSynthesizer:** Uses Probabilistic AutoRegressive models to model sequential dependencies in the data.
+
+*There are more synthesizers, also from SDV, but not all open source.* We used the first four, when evaluating optimal synthesizer for our different use cases.
+
+**Generators - generating new data - on demand**
+
+Once trained, the synthesizer uses the learned model to generate new synthetic data that follows the same statistical properties and 
+distributions as the original dataset, without directly copying any real data points. If you need more data? Just call the generator.
+
+### Our on-premise analytics platform - ARCUS
+
+<div style="display: flex; align-items: center; gap: 20px;">
+  <p>
+    ARCUS is Telia’s advanced on-premise analytics platform, designed to support a wide range of use cases.
+    The platform provides a Kubeflow-based MLOps environment for descriptive, predictive, generative, and (ongoing) agentic AI. 
+    Fully built on open-source, ARCUS integrates a comprehensive stack of components into a unified platform - where Kubernetes is the cornerstone.
+  </p>
+</div>
+
+### Needed environment to create synthetic data
+
+For an efficient, automated selection of the best synthesizer, we need a number of things - from the underlying platform with GPUs and MLOps (Kubeflow).
+
+- Kubeflow pipelines
+- GPU capabilities (for performance and efficiency)
+- Development (IDE) environment (for framework building and running)
+- Modern data platform (MinIO, Airflow) automating the synthetic data generation datasets
+
+#### Parallelism needed
+
+In the (Kube)flows below, we run evaluations in parallel - one for respective synthesizer, followed by a comparison of usability and fidelity scores, selecting the 'winner'.
+
+*Note:* In earlier version of Kubeflow we noticed that the parallelism wasn't acting as expected, waiting for all threads to complete before moving to next step. We had to create a temporary workaround for this, now solved in Kubeflow Pipelines 2.3.0.
+
+Below, we briefly describe the base flow for selecting synthesizer, followed by one use case where we use the resulting data generator for ML development in cloud.
+
+## Exploring the creation and usefulness of synthetic data
+
+This is what we want to do: We have a use case, the supporting data, and developed ML model.
+
+- How similar is the synthetic data compared to the real data (interesting for e.g. visualization use cases)?
+- How well do the ML models based on synthetic data keep up with ML models based on real data?
+
+**General validation of synthetic data technique**
+
+1. Create synthetic data, save the best synthetic data generator. In this step similarity measures are created by the used (SDV) framework 'out of the box'
+2. Create ML model (in our case classifier model) both on real data, as for synthetic data. Compare performance of both models against the (same) real data testset.
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: franciscojavierarceo
body: ```suggestion
2. Create the ML model (in our case classifier model) both on real data and the using the synthetic data. Compare the performance of both models against the same real data testset.
```
created_at: 2025-02-24T16:54:38Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,190 @@
+---
+title: "AI-Generated Synthetic Data, using Kubeflow pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic data - why and how?
+
+The best results come from real data, but accessing it often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete.
+
+While the above focuses on speed and augmentation, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is it, how do we evaluate?
+
+### Evaluation criteria for synthetic data - how do we know it's useful?
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthesizer is best for our specific dataset and use case.
+
+We need to ensure a good balance between: 
+
+  - Usability (how useful is the synthetic data for your use case), 
+  - Fidelity (how similar), and 
+  - Privacy (approved level for use case)
+
+For now, we are focusing only on usability and fidelity, using framework-provided measurements for fidelity and workflows described below for usability.
+
+### Methods and frameworks creating synthetic data 
+
+For us, open source frameworks are the only ones of interest. 
+Cloud-based solutions that require sending data samples to the cloud miss the whole point — some of our data cannot be sent to the cloud. 
+For data already in cloud, we can use other cloud-based frameworks.
+Synthesizers are motivated by multiple factors, but in this context, our focus remains on generating synthetic data for on-premise use.
+
+So, what framework did we (initially) choose? Currently we are using the open source version of [SDV](https://sdv.dev/), 
+an easy-to-use framework with a strong community and many useful features out-of-the-box (e.g. built-in evaluators, many modeling techniques). 
+The field of synthetic data is evolving rapidly. While we do not aim to cover the latest advancements exhaustively, the use of foundational models is certainly an area of interest.
+
+**Comments on privacy and privacy preserving techniques**
+
+Ensuring privacy in synthetic data is a non-trivial problem, even if there
+are techniques to ensure levels of privacy, it remains an active area of research. 
+
+*Privacy problems, in synthetic data?*
+
+Yes, when creating synthetic data we build models, models from which we generate synthetic data.
+However, these models can e.g. be overfitted, resulting in leakage of the training data (the real data). 
+Another challenge lies in handling anomalies. If certain information in the real data stands out and we don't account for it when creating synthetic data, there is a risk of backtracking from the synthetic data to the real.
+A typical example could be 'a very rich person in the dataset' or 'a very rare disease'. 
+The problem here, besides from the privacy part, is that it might be the anomalies we really are looking for.
+
+We are experimenting with various differential privacy strategies, but it is still early days, and we do not focus on them in the examples below.
+
+### The Synthetic Data Vault (SDV) - high level
+
+When you initialize and fit a synthesizer (like GaussianCopulaSynthesizer, CTGANSynthesizer, etc.), it trains a model based on 
+the dataset you provide. This model learns the distribution of the data, capturing the relationships and dependencies between 
+different features in the dataset. Below are the (free) synthesizers provided by SDV that we evaluated on each use case.
+
+The synthesizer doesn't memorize individual records from the dataset. 
+Instead, it tries to learn the underlying statistical patterns, correlations, and distributions present in the data. 
+Each synthesizer does this differently:
+
+- **GaussianCopulaSynthesizer:** Models the distribution of each feature using copulas and marginal distributions. 
+- **CTGANSynthesizer:** Uses a GAN (Generative Adversarial Network) approach, which involves training two neural networks (a generator and a discriminator) to model complex data distributions, especially useful for discrete or mixed-type data.
+- **TVAESynthesizer:** Uses a Variational Autoencoder (VAE) approach to learn a lower-dimensional representation of the data and then generate new samples.
+- **CopulaGANSynthesizer:** Combines elements of copulas and GANs to capture complex dependencies.
+- **PARSynthesizer:** Uses Probabilistic AutoRegressive models to model sequential dependencies in the data.
+
+*There are more synthesizers, also from SDV, but not all open source.* We used the first four, when evaluating optimal synthesizer for our different use cases.
+
+**Generators - generating new data - on demand**
+
+Once trained, the synthesizer uses the learned model to generate new synthetic data that follows the same statistical properties and 
+distributions as the original dataset, without directly copying any real data points. If you need more data? Just call the generator.
+
+### Our on-premise analytics platform - ARCUS
+
+<div style="display: flex; align-items: center; gap: 20px;">
+  <p>
+    ARCUS is Telia’s advanced on-premise analytics platform, designed to support a wide range of use cases.
+    The platform provides a Kubeflow-based MLOps environment for descriptive, predictive, generative, and (ongoing) agentic AI. 
+    Fully built on open-source, ARCUS integrates a comprehensive stack of components into a unified platform - where Kubernetes is the cornerstone.
+  </p>
+</div>
+
+### Needed environment to create synthetic data
+
+For an efficient, automated selection of the best synthesizer, we need a number of things - from the underlying platform with GPUs and MLOps (Kubeflow).
+
+- Kubeflow pipelines
+- GPU capabilities (for performance and efficiency)
+- Development (IDE) environment (for framework building and running)
+- Modern data platform (MinIO, Airflow) automating the synthetic data generation datasets
+
+#### Parallelism needed
+
+In the (Kube)flows below, we run evaluations in parallel - one for respective synthesizer, followed by a comparison of usability and fidelity scores, selecting the 'winner'.
+
+*Note:* In earlier version of Kubeflow we noticed that the parallelism wasn't acting as expected, waiting for all threads to complete before moving to next step. We had to create a temporary workaround for this, now solved in Kubeflow Pipelines 2.3.0.
+
+Below, we briefly describe the base flow for selecting synthesizer, followed by one use case where we use the resulting data generator for ML development in cloud.
+
+## Exploring the creation and usefulness of synthetic data
+
+This is what we want to do: We have a use case, the supporting data, and developed ML model.
+
+- How similar is the synthetic data compared to the real data (interesting for e.g. visualization use cases)?
+- How well do the ML models based on synthetic data keep up with ML models based on real data?
+
+**General validation of synthetic data technique**
+
+1. Create synthetic data, save the best synthetic data generator. In this step similarity measures are created by the used (SDV) framework 'out of the box'
+2. Create ML model (in our case classifier model) both on real data, as for synthetic data. Compare performance of both models against the (same) real data testset.
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image-2.png" width="800">
+
+From above, we have an example where the final synthesizer is collected and saved. This step is used in below example, exporting the resulting synthetic data generator to cloud.
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: franciscojavierarceo
body: ```suggestion
From above, we have an example where the final synthesizer is collected and saved. This step is used in the example  below, exporting the resulting synthetic data generator to cloud.
```
created_at: 2025-02-24T16:55:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,190 @@
+---
+title: "AI-Generated Synthetic Data, using Kubeflow pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic data - why and how?
+
+The best results come from real data, but accessing it often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete.
+
+While the above focuses on speed and augmentation, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is it, how do we evaluate?
+
+### Evaluation criteria for synthetic data - how do we know it's useful?
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthesizer is best for our specific dataset and use case.
+
+We need to ensure a good balance between: 
+
+  - Usability (how useful is the synthetic data for your use case), 
+  - Fidelity (how similar), and 
+  - Privacy (approved level for use case)
+
+For now, we are focusing only on usability and fidelity, using framework-provided measurements for fidelity and workflows described below for usability.
+
+### Methods and frameworks creating synthetic data 
+
+For us, open source frameworks are the only ones of interest. 
+Cloud-based solutions that require sending data samples to the cloud miss the whole point — some of our data cannot be sent to the cloud. 
+For data already in cloud, we can use other cloud-based frameworks.
+Synthesizers are motivated by multiple factors, but in this context, our focus remains on generating synthetic data for on-premise use.
+
+So, what framework did we (initially) choose? Currently we are using the open source version of [SDV](https://sdv.dev/), 
+an easy-to-use framework with a strong community and many useful features out-of-the-box (e.g. built-in evaluators, many modeling techniques). 
+The field of synthetic data is evolving rapidly. While we do not aim to cover the latest advancements exhaustively, the use of foundational models is certainly an area of interest.
+
+**Comments on privacy and privacy preserving techniques**
+
+Ensuring privacy in synthetic data is a non-trivial problem, even if there
+are techniques to ensure levels of privacy, it remains an active area of research. 
+
+*Privacy problems, in synthetic data?*
+
+Yes, when creating synthetic data we build models, models from which we generate synthetic data.
+However, these models can e.g. be overfitted, resulting in leakage of the training data (the real data). 
+Another challenge lies in handling anomalies. If certain information in the real data stands out and we don't account for it when creating synthetic data, there is a risk of backtracking from the synthetic data to the real.
+A typical example could be 'a very rich person in the dataset' or 'a very rare disease'. 
+The problem here, besides from the privacy part, is that it might be the anomalies we really are looking for.
+
+We are experimenting with various differential privacy strategies, but it is still early days, and we do not focus on them in the examples below.
+
+### The Synthetic Data Vault (SDV) - high level
+
+When you initialize and fit a synthesizer (like GaussianCopulaSynthesizer, CTGANSynthesizer, etc.), it trains a model based on 
+the dataset you provide. This model learns the distribution of the data, capturing the relationships and dependencies between 
+different features in the dataset. Below are the (free) synthesizers provided by SDV that we evaluated on each use case.
+
+The synthesizer doesn't memorize individual records from the dataset. 
+Instead, it tries to learn the underlying statistical patterns, correlations, and distributions present in the data. 
+Each synthesizer does this differently:
+
+- **GaussianCopulaSynthesizer:** Models the distribution of each feature using copulas and marginal distributions. 
+- **CTGANSynthesizer:** Uses a GAN (Generative Adversarial Network) approach, which involves training two neural networks (a generator and a discriminator) to model complex data distributions, especially useful for discrete or mixed-type data.
+- **TVAESynthesizer:** Uses a Variational Autoencoder (VAE) approach to learn a lower-dimensional representation of the data and then generate new samples.
+- **CopulaGANSynthesizer:** Combines elements of copulas and GANs to capture complex dependencies.
+- **PARSynthesizer:** Uses Probabilistic AutoRegressive models to model sequential dependencies in the data.
+
+*There are more synthesizers, also from SDV, but not all open source.* We used the first four, when evaluating optimal synthesizer for our different use cases.
+
+**Generators - generating new data - on demand**
+
+Once trained, the synthesizer uses the learned model to generate new synthetic data that follows the same statistical properties and 
+distributions as the original dataset, without directly copying any real data points. If you need more data? Just call the generator.
+
+### Our on-premise analytics platform - ARCUS
+
+<div style="display: flex; align-items: center; gap: 20px;">
+  <p>
+    ARCUS is Telia’s advanced on-premise analytics platform, designed to support a wide range of use cases.
+    The platform provides a Kubeflow-based MLOps environment for descriptive, predictive, generative, and (ongoing) agentic AI. 
+    Fully built on open-source, ARCUS integrates a comprehensive stack of components into a unified platform - where Kubernetes is the cornerstone.
+  </p>
+</div>
+
+### Needed environment to create synthetic data
+
+For an efficient, automated selection of the best synthesizer, we need a number of things - from the underlying platform with GPUs and MLOps (Kubeflow).
+
+- Kubeflow pipelines
+- GPU capabilities (for performance and efficiency)
+- Development (IDE) environment (for framework building and running)
+- Modern data platform (MinIO, Airflow) automating the synthetic data generation datasets
+
+#### Parallelism needed
+
+In the (Kube)flows below, we run evaluations in parallel - one for respective synthesizer, followed by a comparison of usability and fidelity scores, selecting the 'winner'.
+
+*Note:* In earlier version of Kubeflow we noticed that the parallelism wasn't acting as expected, waiting for all threads to complete before moving to next step. We had to create a temporary workaround for this, now solved in Kubeflow Pipelines 2.3.0.
+
+Below, we briefly describe the base flow for selecting synthesizer, followed by one use case where we use the resulting data generator for ML development in cloud.
+
+## Exploring the creation and usefulness of synthetic data
+
+This is what we want to do: We have a use case, the supporting data, and developed ML model.
+
+- How similar is the synthetic data compared to the real data (interesting for e.g. visualization use cases)?
+- How well do the ML models based on synthetic data keep up with ML models based on real data?
+
+**General validation of synthetic data technique**
+
+1. Create synthetic data, save the best synthetic data generator. In this step similarity measures are created by the used (SDV) framework 'out of the box'
+2. Create ML model (in our case classifier model) both on real data, as for synthetic data. Compare performance of both models against the (same) real data testset.
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image-2.png" width="800">
+
+From above, we have an example where the final synthesizer is collected and saved. This step is used in below example, exporting the resulting synthetic data generator to cloud.
+
+## Using synthetic data generators, to enable multiple environments without data transfer
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: franciscojavierarceo
body: ```suggestion
## Using Synthetic Data Generators to Enable Multiple Environments without Data Transfer
```
created_at: 2025-02-24T16:55:25Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,190 @@
+---
+title: "AI-Generated Synthetic Data, using Kubeflow pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic data - why and how?
+
+The best results come from real data, but accessing it often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete.
+
+While the above focuses on speed and augmentation, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is it, how do we evaluate?
+
+### Evaluation criteria for synthetic data - how do we know it's useful?
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthesizer is best for our specific dataset and use case.
+
+We need to ensure a good balance between: 
+
+  - Usability (how useful is the synthetic data for your use case), 
+  - Fidelity (how similar), and 
+  - Privacy (approved level for use case)
+
+For now, we are focusing only on usability and fidelity, using framework-provided measurements for fidelity and workflows described below for usability.
+
+### Methods and frameworks creating synthetic data 
+
+For us, open source frameworks are the only ones of interest. 
+Cloud-based solutions that require sending data samples to the cloud miss the whole point — some of our data cannot be sent to the cloud. 
+For data already in cloud, we can use other cloud-based frameworks.
+Synthesizers are motivated by multiple factors, but in this context, our focus remains on generating synthetic data for on-premise use.
+
+So, what framework did we (initially) choose? Currently we are using the open source version of [SDV](https://sdv.dev/), 
+an easy-to-use framework with a strong community and many useful features out-of-the-box (e.g. built-in evaluators, many modeling techniques). 
+The field of synthetic data is evolving rapidly. While we do not aim to cover the latest advancements exhaustively, the use of foundational models is certainly an area of interest.
+
+**Comments on privacy and privacy preserving techniques**
+
+Ensuring privacy in synthetic data is a non-trivial problem, even if there
+are techniques to ensure levels of privacy, it remains an active area of research. 
+
+*Privacy problems, in synthetic data?*
+
+Yes, when creating synthetic data we build models, models from which we generate synthetic data.
+However, these models can e.g. be overfitted, resulting in leakage of the training data (the real data). 
+Another challenge lies in handling anomalies. If certain information in the real data stands out and we don't account for it when creating synthetic data, there is a risk of backtracking from the synthetic data to the real.
+A typical example could be 'a very rich person in the dataset' or 'a very rare disease'. 
+The problem here, besides from the privacy part, is that it might be the anomalies we really are looking for.
+
+We are experimenting with various differential privacy strategies, but it is still early days, and we do not focus on them in the examples below.
+
+### The Synthetic Data Vault (SDV) - high level
+
+When you initialize and fit a synthesizer (like GaussianCopulaSynthesizer, CTGANSynthesizer, etc.), it trains a model based on 
+the dataset you provide. This model learns the distribution of the data, capturing the relationships and dependencies between 
+different features in the dataset. Below are the (free) synthesizers provided by SDV that we evaluated on each use case.
+
+The synthesizer doesn't memorize individual records from the dataset. 
+Instead, it tries to learn the underlying statistical patterns, correlations, and distributions present in the data. 
+Each synthesizer does this differently:
+
+- **GaussianCopulaSynthesizer:** Models the distribution of each feature using copulas and marginal distributions. 
+- **CTGANSynthesizer:** Uses a GAN (Generative Adversarial Network) approach, which involves training two neural networks (a generator and a discriminator) to model complex data distributions, especially useful for discrete or mixed-type data.
+- **TVAESynthesizer:** Uses a Variational Autoencoder (VAE) approach to learn a lower-dimensional representation of the data and then generate new samples.
+- **CopulaGANSynthesizer:** Combines elements of copulas and GANs to capture complex dependencies.
+- **PARSynthesizer:** Uses Probabilistic AutoRegressive models to model sequential dependencies in the data.
+
+*There are more synthesizers, also from SDV, but not all open source.* We used the first four, when evaluating optimal synthesizer for our different use cases.
+
+**Generators - generating new data - on demand**
+
+Once trained, the synthesizer uses the learned model to generate new synthetic data that follows the same statistical properties and 
+distributions as the original dataset, without directly copying any real data points. If you need more data? Just call the generator.
+
+### Our on-premise analytics platform - ARCUS
+
+<div style="display: flex; align-items: center; gap: 20px;">
+  <p>
+    ARCUS is Telia’s advanced on-premise analytics platform, designed to support a wide range of use cases.
+    The platform provides a Kubeflow-based MLOps environment for descriptive, predictive, generative, and (ongoing) agentic AI. 
+    Fully built on open-source, ARCUS integrates a comprehensive stack of components into a unified platform - where Kubernetes is the cornerstone.
+  </p>
+</div>
+
+### Needed environment to create synthetic data
+
+For an efficient, automated selection of the best synthesizer, we need a number of things - from the underlying platform with GPUs and MLOps (Kubeflow).
+
+- Kubeflow pipelines
+- GPU capabilities (for performance and efficiency)
+- Development (IDE) environment (for framework building and running)
+- Modern data platform (MinIO, Airflow) automating the synthetic data generation datasets
+
+#### Parallelism needed
+
+In the (Kube)flows below, we run evaluations in parallel - one for respective synthesizer, followed by a comparison of usability and fidelity scores, selecting the 'winner'.
+
+*Note:* In earlier version of Kubeflow we noticed that the parallelism wasn't acting as expected, waiting for all threads to complete before moving to next step. We had to create a temporary workaround for this, now solved in Kubeflow Pipelines 2.3.0.
+
+Below, we briefly describe the base flow for selecting synthesizer, followed by one use case where we use the resulting data generator for ML development in cloud.
+
+## Exploring the creation and usefulness of synthetic data
+
+This is what we want to do: We have a use case, the supporting data, and developed ML model.
+
+- How similar is the synthetic data compared to the real data (interesting for e.g. visualization use cases)?
+- How well do the ML models based on synthetic data keep up with ML models based on real data?
+
+**General validation of synthetic data technique**
+
+1. Create synthetic data, save the best synthetic data generator. In this step similarity measures are created by the used (SDV) framework 'out of the box'
+2. Create ML model (in our case classifier model) both on real data, as for synthetic data. Compare performance of both models against the (same) real data testset.
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image-2.png" width="800">
+
+From above, we have an example where the final synthesizer is collected and saved. This step is used in below example, exporting the resulting synthetic data generator to cloud.
+
+## Using synthetic data generators, to enable multiple environments without data transfer
+
+*Below is a usecase where we need to make use of both on-premise and cloud, without moving data to cloud.*
+
+**Problem statement:** 
+
+- Our data cannot be moved from on-premise to cloud.
+- We need extra compute power, in our public cloud environment, to create an ML model for use on-premise.
+- The ML model is to be used on-premise, on new incoming data streams (that cannot be moved to cloud)
+
+**Solution:**
+
+1. Create synthetic data for our on-premise environment use-cases, and - as a side product we save 
+away the synthetic data generator (the pickled model used to create synthetic data). 
+2. Copy the synthetic data generator to cloud
+3. Use the synthetic data generator in cloud, creating synthetic data for training of ML model
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: franciscojavierarceo
body: ```suggestion
3. Use the synthetic data generator in the cloud, creating synthetic data for training of an ML model
```
created_at: 2025-02-24T16:57:11Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,190 @@
+---
+title: "AI-Generated Synthetic Data, using Kubeflow pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic data - why and how?
+
+The best results come from real data, but accessing it often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete.
+
+While the above focuses on speed and augmentation, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is it, how do we evaluate?
+
+### Evaluation criteria for synthetic data - how do we know it's useful?
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthesizer is best for our specific dataset and use case.
+
+We need to ensure a good balance between: 
+
+  - Usability (how useful is the synthetic data for your use case), 
+  - Fidelity (how similar), and 
+  - Privacy (approved level for use case)
+
+For now, we are focusing only on usability and fidelity, using framework-provided measurements for fidelity and workflows described below for usability.
+
+### Methods and frameworks creating synthetic data 
+
+For us, open source frameworks are the only ones of interest. 
+Cloud-based solutions that require sending data samples to the cloud miss the whole point — some of our data cannot be sent to the cloud. 
+For data already in cloud, we can use other cloud-based frameworks.
+Synthesizers are motivated by multiple factors, but in this context, our focus remains on generating synthetic data for on-premise use.
+
+So, what framework did we (initially) choose? Currently we are using the open source version of [SDV](https://sdv.dev/), 
+an easy-to-use framework with a strong community and many useful features out-of-the-box (e.g. built-in evaluators, many modeling techniques). 
+The field of synthetic data is evolving rapidly. While we do not aim to cover the latest advancements exhaustively, the use of foundational models is certainly an area of interest.
+
+**Comments on privacy and privacy preserving techniques**
+
+Ensuring privacy in synthetic data is a non-trivial problem, even if there
+are techniques to ensure levels of privacy, it remains an active area of research. 
+
+*Privacy problems, in synthetic data?*
+
+Yes, when creating synthetic data we build models, models from which we generate synthetic data.
+However, these models can e.g. be overfitted, resulting in leakage of the training data (the real data). 
+Another challenge lies in handling anomalies. If certain information in the real data stands out and we don't account for it when creating synthetic data, there is a risk of backtracking from the synthetic data to the real.
+A typical example could be 'a very rich person in the dataset' or 'a very rare disease'. 
+The problem here, besides from the privacy part, is that it might be the anomalies we really are looking for.
+
+We are experimenting with various differential privacy strategies, but it is still early days, and we do not focus on them in the examples below.
+
+### The Synthetic Data Vault (SDV) - high level
+
+When you initialize and fit a synthesizer (like GaussianCopulaSynthesizer, CTGANSynthesizer, etc.), it trains a model based on 
+the dataset you provide. This model learns the distribution of the data, capturing the relationships and dependencies between 
+different features in the dataset. Below are the (free) synthesizers provided by SDV that we evaluated on each use case.
+
+The synthesizer doesn't memorize individual records from the dataset. 
+Instead, it tries to learn the underlying statistical patterns, correlations, and distributions present in the data. 
+Each synthesizer does this differently:
+
+- **GaussianCopulaSynthesizer:** Models the distribution of each feature using copulas and marginal distributions. 
+- **CTGANSynthesizer:** Uses a GAN (Generative Adversarial Network) approach, which involves training two neural networks (a generator and a discriminator) to model complex data distributions, especially useful for discrete or mixed-type data.
+- **TVAESynthesizer:** Uses a Variational Autoencoder (VAE) approach to learn a lower-dimensional representation of the data and then generate new samples.
+- **CopulaGANSynthesizer:** Combines elements of copulas and GANs to capture complex dependencies.
+- **PARSynthesizer:** Uses Probabilistic AutoRegressive models to model sequential dependencies in the data.
+
+*There are more synthesizers, also from SDV, but not all open source.* We used the first four, when evaluating optimal synthesizer for our different use cases.
+
+**Generators - generating new data - on demand**
+
+Once trained, the synthesizer uses the learned model to generate new synthetic data that follows the same statistical properties and 
+distributions as the original dataset, without directly copying any real data points. If you need more data? Just call the generator.
+
+### Our on-premise analytics platform - ARCUS
+
+<div style="display: flex; align-items: center; gap: 20px;">
+  <p>
+    ARCUS is Telia’s advanced on-premise analytics platform, designed to support a wide range of use cases.
+    The platform provides a Kubeflow-based MLOps environment for descriptive, predictive, generative, and (ongoing) agentic AI. 
+    Fully built on open-source, ARCUS integrates a comprehensive stack of components into a unified platform - where Kubernetes is the cornerstone.
+  </p>
+</div>
+
+### Needed environment to create synthetic data
+
+For an efficient, automated selection of the best synthesizer, we need a number of things - from the underlying platform with GPUs and MLOps (Kubeflow).
+
+- Kubeflow pipelines
+- GPU capabilities (for performance and efficiency)
+- Development (IDE) environment (for framework building and running)
+- Modern data platform (MinIO, Airflow) automating the synthetic data generation datasets
+
+#### Parallelism needed
+
+In the (Kube)flows below, we run evaluations in parallel - one for respective synthesizer, followed by a comparison of usability and fidelity scores, selecting the 'winner'.
+
+*Note:* In earlier version of Kubeflow we noticed that the parallelism wasn't acting as expected, waiting for all threads to complete before moving to next step. We had to create a temporary workaround for this, now solved in Kubeflow Pipelines 2.3.0.
+
+Below, we briefly describe the base flow for selecting synthesizer, followed by one use case where we use the resulting data generator for ML development in cloud.
+
+## Exploring the creation and usefulness of synthetic data
+
+This is what we want to do: We have a use case, the supporting data, and developed ML model.
+
+- How similar is the synthetic data compared to the real data (interesting for e.g. visualization use cases)?
+- How well do the ML models based on synthetic data keep up with ML models based on real data?
+
+**General validation of synthetic data technique**
+
+1. Create synthetic data, save the best synthetic data generator. In this step similarity measures are created by the used (SDV) framework 'out of the box'
+2. Create ML model (in our case classifier model) both on real data, as for synthetic data. Compare performance of both models against the (same) real data testset.
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image-2.png" width="800">
+
+From above, we have an example where the final synthesizer is collected and saved. This step is used in below example, exporting the resulting synthetic data generator to cloud.
+
+## Using synthetic data generators, to enable multiple environments without data transfer
+
+*Below is a usecase where we need to make use of both on-premise and cloud, without moving data to cloud.*
+
+**Problem statement:** 
+
+- Our data cannot be moved from on-premise to cloud.
+- We need extra compute power, in our public cloud environment, to create an ML model for use on-premise.
+- The ML model is to be used on-premise, on new incoming data streams (that cannot be moved to cloud)
+
+**Solution:**
+
+1. Create synthetic data for our on-premise environment use-cases, and - as a side product we save 
+away the synthetic data generator (the pickled model used to create synthetic data). 
+2. Copy the synthetic data generator to cloud
+3. Use the synthetic data generator in cloud, creating synthetic data for training of ML model
+4. Copy ML model to on-premise, and use it for new incoming data
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: franciscojavierarceo
body: ```suggestion
4. Copy the ML model on-premise, and use it for new incoming data
```
created_at: 2025-02-24T16:57:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,190 @@
+---
+title: "AI-Generated Synthetic Data, using Kubeflow pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic data - why and how?
+
+The best results come from real data, but accessing it often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete.
+
+While the above focuses on speed and augmentation, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is it, how do we evaluate?
+
+### Evaluation criteria for synthetic data - how do we know it's useful?
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthesizer is best for our specific dataset and use case.
+
+We need to ensure a good balance between: 
+
+  - Usability (how useful is the synthetic data for your use case), 
+  - Fidelity (how similar), and 
+  - Privacy (approved level for use case)
+
+For now, we are focusing only on usability and fidelity, using framework-provided measurements for fidelity and workflows described below for usability.
+
+### Methods and frameworks creating synthetic data 
+
+For us, open source frameworks are the only ones of interest. 
+Cloud-based solutions that require sending data samples to the cloud miss the whole point — some of our data cannot be sent to the cloud. 
+For data already in cloud, we can use other cloud-based frameworks.
+Synthesizers are motivated by multiple factors, but in this context, our focus remains on generating synthetic data for on-premise use.
+
+So, what framework did we (initially) choose? Currently we are using the open source version of [SDV](https://sdv.dev/), 
+an easy-to-use framework with a strong community and many useful features out-of-the-box (e.g. built-in evaluators, many modeling techniques). 
+The field of synthetic data is evolving rapidly. While we do not aim to cover the latest advancements exhaustively, the use of foundational models is certainly an area of interest.
+
+**Comments on privacy and privacy preserving techniques**
+
+Ensuring privacy in synthetic data is a non-trivial problem, even if there
+are techniques to ensure levels of privacy, it remains an active area of research. 
+
+*Privacy problems, in synthetic data?*
+
+Yes, when creating synthetic data we build models, models from which we generate synthetic data.
+However, these models can e.g. be overfitted, resulting in leakage of the training data (the real data). 
+Another challenge lies in handling anomalies. If certain information in the real data stands out and we don't account for it when creating synthetic data, there is a risk of backtracking from the synthetic data to the real.
+A typical example could be 'a very rich person in the dataset' or 'a very rare disease'. 
+The problem here, besides from the privacy part, is that it might be the anomalies we really are looking for.
+
+We are experimenting with various differential privacy strategies, but it is still early days, and we do not focus on them in the examples below.
+
+### The Synthetic Data Vault (SDV) - high level
+
+When you initialize and fit a synthesizer (like GaussianCopulaSynthesizer, CTGANSynthesizer, etc.), it trains a model based on 
+the dataset you provide. This model learns the distribution of the data, capturing the relationships and dependencies between 
+different features in the dataset. Below are the (free) synthesizers provided by SDV that we evaluated on each use case.
+
+The synthesizer doesn't memorize individual records from the dataset. 
+Instead, it tries to learn the underlying statistical patterns, correlations, and distributions present in the data. 
+Each synthesizer does this differently:
+
+- **GaussianCopulaSynthesizer:** Models the distribution of each feature using copulas and marginal distributions. 
+- **CTGANSynthesizer:** Uses a GAN (Generative Adversarial Network) approach, which involves training two neural networks (a generator and a discriminator) to model complex data distributions, especially useful for discrete or mixed-type data.
+- **TVAESynthesizer:** Uses a Variational Autoencoder (VAE) approach to learn a lower-dimensional representation of the data and then generate new samples.
+- **CopulaGANSynthesizer:** Combines elements of copulas and GANs to capture complex dependencies.
+- **PARSynthesizer:** Uses Probabilistic AutoRegressive models to model sequential dependencies in the data.
+
+*There are more synthesizers, also from SDV, but not all open source.* We used the first four, when evaluating optimal synthesizer for our different use cases.
+
+**Generators - generating new data - on demand**
+
+Once trained, the synthesizer uses the learned model to generate new synthetic data that follows the same statistical properties and 
+distributions as the original dataset, without directly copying any real data points. If you need more data? Just call the generator.
+
+### Our on-premise analytics platform - ARCUS
+
+<div style="display: flex; align-items: center; gap: 20px;">
+  <p>
+    ARCUS is Telia’s advanced on-premise analytics platform, designed to support a wide range of use cases.
+    The platform provides a Kubeflow-based MLOps environment for descriptive, predictive, generative, and (ongoing) agentic AI. 
+    Fully built on open-source, ARCUS integrates a comprehensive stack of components into a unified platform - where Kubernetes is the cornerstone.
+  </p>
+</div>
+
+### Needed environment to create synthetic data
+
+For an efficient, automated selection of the best synthesizer, we need a number of things - from the underlying platform with GPUs and MLOps (Kubeflow).
+
+- Kubeflow pipelines
+- GPU capabilities (for performance and efficiency)
+- Development (IDE) environment (for framework building and running)
+- Modern data platform (MinIO, Airflow) automating the synthetic data generation datasets
+
+#### Parallelism needed
+
+In the (Kube)flows below, we run evaluations in parallel - one for respective synthesizer, followed by a comparison of usability and fidelity scores, selecting the 'winner'.
+
+*Note:* In earlier version of Kubeflow we noticed that the parallelism wasn't acting as expected, waiting for all threads to complete before moving to next step. We had to create a temporary workaround for this, now solved in Kubeflow Pipelines 2.3.0.
+
+Below, we briefly describe the base flow for selecting synthesizer, followed by one use case where we use the resulting data generator for ML development in cloud.
+
+## Exploring the creation and usefulness of synthetic data
+
+This is what we want to do: We have a use case, the supporting data, and developed ML model.
+
+- How similar is the synthetic data compared to the real data (interesting for e.g. visualization use cases)?
+- How well do the ML models based on synthetic data keep up with ML models based on real data?
+
+**General validation of synthetic data technique**
+
+1. Create synthetic data, save the best synthetic data generator. In this step similarity measures are created by the used (SDV) framework 'out of the box'
+2. Create ML model (in our case classifier model) both on real data, as for synthetic data. Compare performance of both models against the (same) real data testset.
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image-2.png" width="800">
+
+From above, we have an example where the final synthesizer is collected and saved. This step is used in below example, exporting the resulting synthetic data generator to cloud.
+
+## Using synthetic data generators, to enable multiple environments without data transfer
+
+*Below is a usecase where we need to make use of both on-premise and cloud, without moving data to cloud.*
+
+**Problem statement:** 
+
+- Our data cannot be moved from on-premise to cloud.
+- We need extra compute power, in our public cloud environment, to create an ML model for use on-premise.
+- The ML model is to be used on-premise, on new incoming data streams (that cannot be moved to cloud)
+
+**Solution:**
+
+1. Create synthetic data for our on-premise environment use-cases, and - as a side product we save 
+away the synthetic data generator (the pickled model used to create synthetic data). 
+2. Copy the synthetic data generator to cloud
+3. Use the synthetic data generator in cloud, creating synthetic data for training of ML model
+4. Copy ML model to on-premise, and use it for new incoming data
+5. Evaluate: Compare the on-premise AI model with the model created in cloud - against same test data
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: franciscojavierarceo
body: ```suggestion
5. Evaluate: Compare the on-premise AI model with the model created in the cloud - against the same test data
```
created_at: 2025-02-24T16:57:51Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,190 @@
+---
+title: "AI-Generated Synthetic Data, using Kubeflow pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic data - why and how?
+
+The best results come from real data, but accessing it often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete.
+
+While the above focuses on speed and augmentation, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is it, how do we evaluate?
+
+### Evaluation criteria for synthetic data - how do we know it's useful?
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthesizer is best for our specific dataset and use case.
+
+We need to ensure a good balance between: 
+
+  - Usability (how useful is the synthetic data for your use case), 
+  - Fidelity (how similar), and 
+  - Privacy (approved level for use case)
+
+For now, we are focusing only on usability and fidelity, using framework-provided measurements for fidelity and workflows described below for usability.
+
+### Methods and frameworks creating synthetic data 
+
+For us, open source frameworks are the only ones of interest. 
+Cloud-based solutions that require sending data samples to the cloud miss the whole point — some of our data cannot be sent to the cloud. 
+For data already in cloud, we can use other cloud-based frameworks.
+Synthesizers are motivated by multiple factors, but in this context, our focus remains on generating synthetic data for on-premise use.
+
+So, what framework did we (initially) choose? Currently we are using the open source version of [SDV](https://sdv.dev/), 
+an easy-to-use framework with a strong community and many useful features out-of-the-box (e.g. built-in evaluators, many modeling techniques). 
+The field of synthetic data is evolving rapidly. While we do not aim to cover the latest advancements exhaustively, the use of foundational models is certainly an area of interest.
+
+**Comments on privacy and privacy preserving techniques**
+
+Ensuring privacy in synthetic data is a non-trivial problem, even if there
+are techniques to ensure levels of privacy, it remains an active area of research. 
+
+*Privacy problems, in synthetic data?*
+
+Yes, when creating synthetic data we build models, models from which we generate synthetic data.
+However, these models can e.g. be overfitted, resulting in leakage of the training data (the real data). 
+Another challenge lies in handling anomalies. If certain information in the real data stands out and we don't account for it when creating synthetic data, there is a risk of backtracking from the synthetic data to the real.
+A typical example could be 'a very rich person in the dataset' or 'a very rare disease'. 
+The problem here, besides from the privacy part, is that it might be the anomalies we really are looking for.
+
+We are experimenting with various differential privacy strategies, but it is still early days, and we do not focus on them in the examples below.
+
+### The Synthetic Data Vault (SDV) - high level
+
+When you initialize and fit a synthesizer (like GaussianCopulaSynthesizer, CTGANSynthesizer, etc.), it trains a model based on 
+the dataset you provide. This model learns the distribution of the data, capturing the relationships and dependencies between 
+different features in the dataset. Below are the (free) synthesizers provided by SDV that we evaluated on each use case.
+
+The synthesizer doesn't memorize individual records from the dataset. 
+Instead, it tries to learn the underlying statistical patterns, correlations, and distributions present in the data. 
+Each synthesizer does this differently:
+
+- **GaussianCopulaSynthesizer:** Models the distribution of each feature using copulas and marginal distributions. 
+- **CTGANSynthesizer:** Uses a GAN (Generative Adversarial Network) approach, which involves training two neural networks (a generator and a discriminator) to model complex data distributions, especially useful for discrete or mixed-type data.
+- **TVAESynthesizer:** Uses a Variational Autoencoder (VAE) approach to learn a lower-dimensional representation of the data and then generate new samples.
+- **CopulaGANSynthesizer:** Combines elements of copulas and GANs to capture complex dependencies.
+- **PARSynthesizer:** Uses Probabilistic AutoRegressive models to model sequential dependencies in the data.
+
+*There are more synthesizers, also from SDV, but not all open source.* We used the first four, when evaluating optimal synthesizer for our different use cases.
+
+**Generators - generating new data - on demand**
+
+Once trained, the synthesizer uses the learned model to generate new synthetic data that follows the same statistical properties and 
+distributions as the original dataset, without directly copying any real data points. If you need more data? Just call the generator.
+
+### Our on-premise analytics platform - ARCUS
+
+<div style="display: flex; align-items: center; gap: 20px;">
+  <p>
+    ARCUS is Telia’s advanced on-premise analytics platform, designed to support a wide range of use cases.
+    The platform provides a Kubeflow-based MLOps environment for descriptive, predictive, generative, and (ongoing) agentic AI. 
+    Fully built on open-source, ARCUS integrates a comprehensive stack of components into a unified platform - where Kubernetes is the cornerstone.
+  </p>
+</div>
+
+### Needed environment to create synthetic data
+
+For an efficient, automated selection of the best synthesizer, we need a number of things - from the underlying platform with GPUs and MLOps (Kubeflow).
+
+- Kubeflow pipelines
+- GPU capabilities (for performance and efficiency)
+- Development (IDE) environment (for framework building and running)
+- Modern data platform (MinIO, Airflow) automating the synthetic data generation datasets
+
+#### Parallelism needed
+
+In the (Kube)flows below, we run evaluations in parallel - one for respective synthesizer, followed by a comparison of usability and fidelity scores, selecting the 'winner'.
+
+*Note:* In earlier version of Kubeflow we noticed that the parallelism wasn't acting as expected, waiting for all threads to complete before moving to next step. We had to create a temporary workaround for this, now solved in Kubeflow Pipelines 2.3.0.
+
+Below, we briefly describe the base flow for selecting synthesizer, followed by one use case where we use the resulting data generator for ML development in cloud.
+
+## Exploring the creation and usefulness of synthetic data
+
+This is what we want to do: We have a use case, the supporting data, and developed ML model.
+
+- How similar is the synthetic data compared to the real data (interesting for e.g. visualization use cases)?
+- How well do the ML models based on synthetic data keep up with ML models based on real data?
+
+**General validation of synthetic data technique**
+
+1. Create synthetic data, save the best synthetic data generator. In this step similarity measures are created by the used (SDV) framework 'out of the box'
+2. Create ML model (in our case classifier model) both on real data, as for synthetic data. Compare performance of both models against the (same) real data testset.
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image-2.png" width="800">
+
+From above, we have an example where the final synthesizer is collected and saved. This step is used in below example, exporting the resulting synthetic data generator to cloud.
+
+## Using synthetic data generators, to enable multiple environments without data transfer
+
+*Below is a usecase where we need to make use of both on-premise and cloud, without moving data to cloud.*
+
+**Problem statement:** 
+
+- Our data cannot be moved from on-premise to cloud.
+- We need extra compute power, in our public cloud environment, to create an ML model for use on-premise.
+- The ML model is to be used on-premise, on new incoming data streams (that cannot be moved to cloud)
+
+**Solution:**
+
+1. Create synthetic data for our on-premise environment use-cases, and - as a side product we save 
+away the synthetic data generator (the pickled model used to create synthetic data). 
+2. Copy the synthetic data generator to cloud
+3. Use the synthetic data generator in cloud, creating synthetic data for training of ML model
+4. Copy ML model to on-premise, and use it for new incoming data
+5. Evaluate: Compare the on-premise AI model with the model created in cloud - against same test data
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image.png" width="700">
+
+
+**Division of work, what is done on-premise Kubeflow, and what is done in cloud (AWS SageMaker)?**
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: franciscojavierarceo
body: ```suggestion
**Division of work, what is done on-premise with Kubeflow, and what is done in cloud (AWS SageMaker)?**
```
created_at: 2025-02-24T16:58:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,190 @@
+---
+title: "AI-Generated Synthetic Data, using Kubeflow pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic data - why and how?
+
+The best results come from real data, but accessing it often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete.
+
+While the above focuses on speed and augmentation, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is it, how do we evaluate?
+
+### Evaluation criteria for synthetic data - how do we know it's useful?
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthesizer is best for our specific dataset and use case.
+
+We need to ensure a good balance between: 
+
+  - Usability (how useful is the synthetic data for your use case), 
+  - Fidelity (how similar), and 
+  - Privacy (approved level for use case)
+
+For now, we are focusing only on usability and fidelity, using framework-provided measurements for fidelity and workflows described below for usability.
+
+### Methods and frameworks creating synthetic data 
+
+For us, open source frameworks are the only ones of interest. 
+Cloud-based solutions that require sending data samples to the cloud miss the whole point — some of our data cannot be sent to the cloud. 
+For data already in cloud, we can use other cloud-based frameworks.
+Synthesizers are motivated by multiple factors, but in this context, our focus remains on generating synthetic data for on-premise use.
+
+So, what framework did we (initially) choose? Currently we are using the open source version of [SDV](https://sdv.dev/), 
+an easy-to-use framework with a strong community and many useful features out-of-the-box (e.g. built-in evaluators, many modeling techniques). 
+The field of synthetic data is evolving rapidly. While we do not aim to cover the latest advancements exhaustively, the use of foundational models is certainly an area of interest.
+
+**Comments on privacy and privacy preserving techniques**
+
+Ensuring privacy in synthetic data is a non-trivial problem, even if there
+are techniques to ensure levels of privacy, it remains an active area of research. 
+
+*Privacy problems, in synthetic data?*
+
+Yes, when creating synthetic data we build models, models from which we generate synthetic data.
+However, these models can e.g. be overfitted, resulting in leakage of the training data (the real data). 
+Another challenge lies in handling anomalies. If certain information in the real data stands out and we don't account for it when creating synthetic data, there is a risk of backtracking from the synthetic data to the real.
+A typical example could be 'a very rich person in the dataset' or 'a very rare disease'. 
+The problem here, besides from the privacy part, is that it might be the anomalies we really are looking for.
+
+We are experimenting with various differential privacy strategies, but it is still early days, and we do not focus on them in the examples below.
+
+### The Synthetic Data Vault (SDV) - high level
+
+When you initialize and fit a synthesizer (like GaussianCopulaSynthesizer, CTGANSynthesizer, etc.), it trains a model based on 
+the dataset you provide. This model learns the distribution of the data, capturing the relationships and dependencies between 
+different features in the dataset. Below are the (free) synthesizers provided by SDV that we evaluated on each use case.
+
+The synthesizer doesn't memorize individual records from the dataset. 
+Instead, it tries to learn the underlying statistical patterns, correlations, and distributions present in the data. 
+Each synthesizer does this differently:
+
+- **GaussianCopulaSynthesizer:** Models the distribution of each feature using copulas and marginal distributions. 
+- **CTGANSynthesizer:** Uses a GAN (Generative Adversarial Network) approach, which involves training two neural networks (a generator and a discriminator) to model complex data distributions, especially useful for discrete or mixed-type data.
+- **TVAESynthesizer:** Uses a Variational Autoencoder (VAE) approach to learn a lower-dimensional representation of the data and then generate new samples.
+- **CopulaGANSynthesizer:** Combines elements of copulas and GANs to capture complex dependencies.
+- **PARSynthesizer:** Uses Probabilistic AutoRegressive models to model sequential dependencies in the data.
+
+*There are more synthesizers, also from SDV, but not all open source.* We used the first four, when evaluating optimal synthesizer for our different use cases.
+
+**Generators - generating new data - on demand**
+
+Once trained, the synthesizer uses the learned model to generate new synthetic data that follows the same statistical properties and 
+distributions as the original dataset, without directly copying any real data points. If you need more data? Just call the generator.
+
+### Our on-premise analytics platform - ARCUS
+
+<div style="display: flex; align-items: center; gap: 20px;">
+  <p>
+    ARCUS is Telia’s advanced on-premise analytics platform, designed to support a wide range of use cases.
+    The platform provides a Kubeflow-based MLOps environment for descriptive, predictive, generative, and (ongoing) agentic AI. 
+    Fully built on open-source, ARCUS integrates a comprehensive stack of components into a unified platform - where Kubernetes is the cornerstone.
+  </p>
+</div>
+
+### Needed environment to create synthetic data
+
+For an efficient, automated selection of the best synthesizer, we need a number of things - from the underlying platform with GPUs and MLOps (Kubeflow).
+
+- Kubeflow pipelines
+- GPU capabilities (for performance and efficiency)
+- Development (IDE) environment (for framework building and running)
+- Modern data platform (MinIO, Airflow) automating the synthetic data generation datasets
+
+#### Parallelism needed
+
+In the (Kube)flows below, we run evaluations in parallel - one for respective synthesizer, followed by a comparison of usability and fidelity scores, selecting the 'winner'.
+
+*Note:* In earlier version of Kubeflow we noticed that the parallelism wasn't acting as expected, waiting for all threads to complete before moving to next step. We had to create a temporary workaround for this, now solved in Kubeflow Pipelines 2.3.0.
+
+Below, we briefly describe the base flow for selecting synthesizer, followed by one use case where we use the resulting data generator for ML development in cloud.
+
+## Exploring the creation and usefulness of synthetic data
+
+This is what we want to do: We have a use case, the supporting data, and developed ML model.
+
+- How similar is the synthetic data compared to the real data (interesting for e.g. visualization use cases)?
+- How well do the ML models based on synthetic data keep up with ML models based on real data?
+
+**General validation of synthetic data technique**
+
+1. Create synthetic data, save the best synthetic data generator. In this step similarity measures are created by the used (SDV) framework 'out of the box'
+2. Create ML model (in our case classifier model) both on real data, as for synthetic data. Compare performance of both models against the (same) real data testset.
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image-2.png" width="800">
+
+From above, we have an example where the final synthesizer is collected and saved. This step is used in below example, exporting the resulting synthetic data generator to cloud.
+
+## Using synthetic data generators, to enable multiple environments without data transfer
+
+*Below is a usecase where we need to make use of both on-premise and cloud, without moving data to cloud.*
+
+**Problem statement:** 
+
+- Our data cannot be moved from on-premise to cloud.
+- We need extra compute power, in our public cloud environment, to create an ML model for use on-premise.
+- The ML model is to be used on-premise, on new incoming data streams (that cannot be moved to cloud)
+
+**Solution:**
+
+1. Create synthetic data for our on-premise environment use-cases, and - as a side product we save 
+away the synthetic data generator (the pickled model used to create synthetic data). 
+2. Copy the synthetic data generator to cloud
+3. Use the synthetic data generator in cloud, creating synthetic data for training of ML model
+4. Copy ML model to on-premise, and use it for new incoming data
+5. Evaluate: Compare the on-premise AI model with the model created in cloud - against same test data
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image.png" width="700">
+
+
+**Division of work, what is done on-premise Kubeflow, and what is done in cloud (AWS SageMaker)?**
+
+##### On-premise
+
+See above *General validation of synthetic data technique*.
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: franciscojavierarceo
body: ```suggestion
See the above *Validation of synthetic data techniques*.
```
created_at: 2025-02-24T16:58:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,190 @@
+---
+title: "AI-Generated Synthetic Data, using Kubeflow pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic data - why and how?
+
+The best results come from real data, but accessing it often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete.
+
+While the above focuses on speed and augmentation, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is it, how do we evaluate?
+
+### Evaluation criteria for synthetic data - how do we know it's useful?
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthesizer is best for our specific dataset and use case.
+
+We need to ensure a good balance between: 
+
+  - Usability (how useful is the synthetic data for your use case), 
+  - Fidelity (how similar), and 
+  - Privacy (approved level for use case)
+
+For now, we are focusing only on usability and fidelity, using framework-provided measurements for fidelity and workflows described below for usability.
+
+### Methods and frameworks creating synthetic data 
+
+For us, open source frameworks are the only ones of interest. 
+Cloud-based solutions that require sending data samples to the cloud miss the whole point — some of our data cannot be sent to the cloud. 
+For data already in cloud, we can use other cloud-based frameworks.
+Synthesizers are motivated by multiple factors, but in this context, our focus remains on generating synthetic data for on-premise use.
+
+So, what framework did we (initially) choose? Currently we are using the open source version of [SDV](https://sdv.dev/), 
+an easy-to-use framework with a strong community and many useful features out-of-the-box (e.g. built-in evaluators, many modeling techniques). 
+The field of synthetic data is evolving rapidly. While we do not aim to cover the latest advancements exhaustively, the use of foundational models is certainly an area of interest.
+
+**Comments on privacy and privacy preserving techniques**
+
+Ensuring privacy in synthetic data is a non-trivial problem, even if there
+are techniques to ensure levels of privacy, it remains an active area of research. 
+
+*Privacy problems, in synthetic data?*
+
+Yes, when creating synthetic data we build models, models from which we generate synthetic data.
+However, these models can e.g. be overfitted, resulting in leakage of the training data (the real data). 
+Another challenge lies in handling anomalies. If certain information in the real data stands out and we don't account for it when creating synthetic data, there is a risk of backtracking from the synthetic data to the real.
+A typical example could be 'a very rich person in the dataset' or 'a very rare disease'. 
+The problem here, besides from the privacy part, is that it might be the anomalies we really are looking for.
+
+We are experimenting with various differential privacy strategies, but it is still early days, and we do not focus on them in the examples below.
+
+### The Synthetic Data Vault (SDV) - high level
+
+When you initialize and fit a synthesizer (like GaussianCopulaSynthesizer, CTGANSynthesizer, etc.), it trains a model based on 
+the dataset you provide. This model learns the distribution of the data, capturing the relationships and dependencies between 
+different features in the dataset. Below are the (free) synthesizers provided by SDV that we evaluated on each use case.
+
+The synthesizer doesn't memorize individual records from the dataset. 
+Instead, it tries to learn the underlying statistical patterns, correlations, and distributions present in the data. 
+Each synthesizer does this differently:
+
+- **GaussianCopulaSynthesizer:** Models the distribution of each feature using copulas and marginal distributions. 
+- **CTGANSynthesizer:** Uses a GAN (Generative Adversarial Network) approach, which involves training two neural networks (a generator and a discriminator) to model complex data distributions, especially useful for discrete or mixed-type data.
+- **TVAESynthesizer:** Uses a Variational Autoencoder (VAE) approach to learn a lower-dimensional representation of the data and then generate new samples.
+- **CopulaGANSynthesizer:** Combines elements of copulas and GANs to capture complex dependencies.
+- **PARSynthesizer:** Uses Probabilistic AutoRegressive models to model sequential dependencies in the data.
+
+*There are more synthesizers, also from SDV, but not all open source.* We used the first four, when evaluating optimal synthesizer for our different use cases.
+
+**Generators - generating new data - on demand**
+
+Once trained, the synthesizer uses the learned model to generate new synthetic data that follows the same statistical properties and 
+distributions as the original dataset, without directly copying any real data points. If you need more data? Just call the generator.
+
+### Our on-premise analytics platform - ARCUS
+
+<div style="display: flex; align-items: center; gap: 20px;">
+  <p>
+    ARCUS is Telia’s advanced on-premise analytics platform, designed to support a wide range of use cases.
+    The platform provides a Kubeflow-based MLOps environment for descriptive, predictive, generative, and (ongoing) agentic AI. 
+    Fully built on open-source, ARCUS integrates a comprehensive stack of components into a unified platform - where Kubernetes is the cornerstone.
+  </p>
+</div>
+
+### Needed environment to create synthetic data
+
+For an efficient, automated selection of the best synthesizer, we need a number of things - from the underlying platform with GPUs and MLOps (Kubeflow).
+
+- Kubeflow pipelines
+- GPU capabilities (for performance and efficiency)
+- Development (IDE) environment (for framework building and running)
+- Modern data platform (MinIO, Airflow) automating the synthetic data generation datasets
+
+#### Parallelism needed
+
+In the (Kube)flows below, we run evaluations in parallel - one for respective synthesizer, followed by a comparison of usability and fidelity scores, selecting the 'winner'.
+
+*Note:* In earlier version of Kubeflow we noticed that the parallelism wasn't acting as expected, waiting for all threads to complete before moving to next step. We had to create a temporary workaround for this, now solved in Kubeflow Pipelines 2.3.0.
+
+Below, we briefly describe the base flow for selecting synthesizer, followed by one use case where we use the resulting data generator for ML development in cloud.
+
+## Exploring the creation and usefulness of synthetic data
+
+This is what we want to do: We have a use case, the supporting data, and developed ML model.
+
+- How similar is the synthetic data compared to the real data (interesting for e.g. visualization use cases)?
+- How well do the ML models based on synthetic data keep up with ML models based on real data?
+
+**General validation of synthetic data technique**
+
+1. Create synthetic data, save the best synthetic data generator. In this step similarity measures are created by the used (SDV) framework 'out of the box'
+2. Create ML model (in our case classifier model) both on real data, as for synthetic data. Compare performance of both models against the (same) real data testset.
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image-2.png" width="800">
+
+From above, we have an example where the final synthesizer is collected and saved. This step is used in below example, exporting the resulting synthetic data generator to cloud.
+
+## Using synthetic data generators, to enable multiple environments without data transfer
+
+*Below is a usecase where we need to make use of both on-premise and cloud, without moving data to cloud.*
+
+**Problem statement:** 
+
+- Our data cannot be moved from on-premise to cloud.
+- We need extra compute power, in our public cloud environment, to create an ML model for use on-premise.
+- The ML model is to be used on-premise, on new incoming data streams (that cannot be moved to cloud)
+
+**Solution:**
+
+1. Create synthetic data for our on-premise environment use-cases, and - as a side product we save 
+away the synthetic data generator (the pickled model used to create synthetic data). 
+2. Copy the synthetic data generator to cloud
+3. Use the synthetic data generator in cloud, creating synthetic data for training of ML model
+4. Copy ML model to on-premise, and use it for new incoming data
+5. Evaluate: Compare the on-premise AI model with the model created in cloud - against same test data
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image.png" width="700">
+
+
+**Division of work, what is done on-premise Kubeflow, and what is done in cloud (AWS SageMaker)?**
+
+##### On-premise
+
+See above *General validation of synthetic data technique*.
+
+- Develop model on real data – for later comparison with cloud model.
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: franciscojavierarceo
body: ```suggestion
- Develop the model on real data – for the comparison later with the cloud model.
```
created_at: 2025-02-24T16:59:07Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,190 @@
+---
+title: "AI-Generated Synthetic Data, using Kubeflow pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic data - why and how?
+
+The best results come from real data, but accessing it often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete.
+
+While the above focuses on speed and augmentation, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is it, how do we evaluate?
+
+### Evaluation criteria for synthetic data - how do we know it's useful?
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthesizer is best for our specific dataset and use case.
+
+We need to ensure a good balance between: 
+
+  - Usability (how useful is the synthetic data for your use case), 
+  - Fidelity (how similar), and 
+  - Privacy (approved level for use case)
+
+For now, we are focusing only on usability and fidelity, using framework-provided measurements for fidelity and workflows described below for usability.
+
+### Methods and frameworks creating synthetic data 
+
+For us, open source frameworks are the only ones of interest. 
+Cloud-based solutions that require sending data samples to the cloud miss the whole point — some of our data cannot be sent to the cloud. 
+For data already in cloud, we can use other cloud-based frameworks.
+Synthesizers are motivated by multiple factors, but in this context, our focus remains on generating synthetic data for on-premise use.
+
+So, what framework did we (initially) choose? Currently we are using the open source version of [SDV](https://sdv.dev/), 
+an easy-to-use framework with a strong community and many useful features out-of-the-box (e.g. built-in evaluators, many modeling techniques). 
+The field of synthetic data is evolving rapidly. While we do not aim to cover the latest advancements exhaustively, the use of foundational models is certainly an area of interest.
+
+**Comments on privacy and privacy preserving techniques**
+
+Ensuring privacy in synthetic data is a non-trivial problem, even if there
+are techniques to ensure levels of privacy, it remains an active area of research. 
+
+*Privacy problems, in synthetic data?*
+
+Yes, when creating synthetic data we build models, models from which we generate synthetic data.
+However, these models can e.g. be overfitted, resulting in leakage of the training data (the real data). 
+Another challenge lies in handling anomalies. If certain information in the real data stands out and we don't account for it when creating synthetic data, there is a risk of backtracking from the synthetic data to the real.
+A typical example could be 'a very rich person in the dataset' or 'a very rare disease'. 
+The problem here, besides from the privacy part, is that it might be the anomalies we really are looking for.
+
+We are experimenting with various differential privacy strategies, but it is still early days, and we do not focus on them in the examples below.
+
+### The Synthetic Data Vault (SDV) - high level
+
+When you initialize and fit a synthesizer (like GaussianCopulaSynthesizer, CTGANSynthesizer, etc.), it trains a model based on 
+the dataset you provide. This model learns the distribution of the data, capturing the relationships and dependencies between 
+different features in the dataset. Below are the (free) synthesizers provided by SDV that we evaluated on each use case.
+
+The synthesizer doesn't memorize individual records from the dataset. 
+Instead, it tries to learn the underlying statistical patterns, correlations, and distributions present in the data. 
+Each synthesizer does this differently:
+
+- **GaussianCopulaSynthesizer:** Models the distribution of each feature using copulas and marginal distributions. 
+- **CTGANSynthesizer:** Uses a GAN (Generative Adversarial Network) approach, which involves training two neural networks (a generator and a discriminator) to model complex data distributions, especially useful for discrete or mixed-type data.
+- **TVAESynthesizer:** Uses a Variational Autoencoder (VAE) approach to learn a lower-dimensional representation of the data and then generate new samples.
+- **CopulaGANSynthesizer:** Combines elements of copulas and GANs to capture complex dependencies.
+- **PARSynthesizer:** Uses Probabilistic AutoRegressive models to model sequential dependencies in the data.
+
+*There are more synthesizers, also from SDV, but not all open source.* We used the first four, when evaluating optimal synthesizer for our different use cases.
+
+**Generators - generating new data - on demand**
+
+Once trained, the synthesizer uses the learned model to generate new synthetic data that follows the same statistical properties and 
+distributions as the original dataset, without directly copying any real data points. If you need more data? Just call the generator.
+
+### Our on-premise analytics platform - ARCUS
+
+<div style="display: flex; align-items: center; gap: 20px;">
+  <p>
+    ARCUS is Telia’s advanced on-premise analytics platform, designed to support a wide range of use cases.
+    The platform provides a Kubeflow-based MLOps environment for descriptive, predictive, generative, and (ongoing) agentic AI. 
+    Fully built on open-source, ARCUS integrates a comprehensive stack of components into a unified platform - where Kubernetes is the cornerstone.
+  </p>
+</div>
+
+### Needed environment to create synthetic data
+
+For an efficient, automated selection of the best synthesizer, we need a number of things - from the underlying platform with GPUs and MLOps (Kubeflow).
+
+- Kubeflow pipelines
+- GPU capabilities (for performance and efficiency)
+- Development (IDE) environment (for framework building and running)
+- Modern data platform (MinIO, Airflow) automating the synthetic data generation datasets
+
+#### Parallelism needed
+
+In the (Kube)flows below, we run evaluations in parallel - one for respective synthesizer, followed by a comparison of usability and fidelity scores, selecting the 'winner'.
+
+*Note:* In earlier version of Kubeflow we noticed that the parallelism wasn't acting as expected, waiting for all threads to complete before moving to next step. We had to create a temporary workaround for this, now solved in Kubeflow Pipelines 2.3.0.
+
+Below, we briefly describe the base flow for selecting synthesizer, followed by one use case where we use the resulting data generator for ML development in cloud.
+
+## Exploring the creation and usefulness of synthetic data
+
+This is what we want to do: We have a use case, the supporting data, and developed ML model.
+
+- How similar is the synthetic data compared to the real data (interesting for e.g. visualization use cases)?
+- How well do the ML models based on synthetic data keep up with ML models based on real data?
+
+**General validation of synthetic data technique**
+
+1. Create synthetic data, save the best synthetic data generator. In this step similarity measures are created by the used (SDV) framework 'out of the box'
+2. Create ML model (in our case classifier model) both on real data, as for synthetic data. Compare performance of both models against the (same) real data testset.
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image-2.png" width="800">
+
+From above, we have an example where the final synthesizer is collected and saved. This step is used in below example, exporting the resulting synthetic data generator to cloud.
+
+## Using synthetic data generators, to enable multiple environments without data transfer
+
+*Below is a usecase where we need to make use of both on-premise and cloud, without moving data to cloud.*
+
+**Problem statement:** 
+
+- Our data cannot be moved from on-premise to cloud.
+- We need extra compute power, in our public cloud environment, to create an ML model for use on-premise.
+- The ML model is to be used on-premise, on new incoming data streams (that cannot be moved to cloud)
+
+**Solution:**
+
+1. Create synthetic data for our on-premise environment use-cases, and - as a side product we save 
+away the synthetic data generator (the pickled model used to create synthetic data). 
+2. Copy the synthetic data generator to cloud
+3. Use the synthetic data generator in cloud, creating synthetic data for training of ML model
+4. Copy ML model to on-premise, and use it for new incoming data
+5. Evaluate: Compare the on-premise AI model with the model created in cloud - against same test data
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image.png" width="700">
+
+
+**Division of work, what is done on-premise Kubeflow, and what is done in cloud (AWS SageMaker)?**
+
+##### On-premise
+
+See above *General validation of synthetic data technique*.
+
+- Develop model on real data – for later comparison with cloud model.
+- Create synthetic generators, evaluate, export best generator to AWS.
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: franciscojavierarceo
body: ```suggestion
- Create synthetic generators, evaluate the generators, and export the best generator to AWS.
```
created_at: 2025-02-24T16:59:34Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,190 @@
+---
+title: "AI-Generated Synthetic Data, using Kubeflow pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic data - why and how?
+
+The best results come from real data, but accessing it often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete.
+
+While the above focuses on speed and augmentation, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is it, how do we evaluate?
+
+### Evaluation criteria for synthetic data - how do we know it's useful?
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthesizer is best for our specific dataset and use case.
+
+We need to ensure a good balance between: 
+
+  - Usability (how useful is the synthetic data for your use case), 
+  - Fidelity (how similar), and 
+  - Privacy (approved level for use case)
+
+For now, we are focusing only on usability and fidelity, using framework-provided measurements for fidelity and workflows described below for usability.
+
+### Methods and frameworks creating synthetic data 
+
+For us, open source frameworks are the only ones of interest. 
+Cloud-based solutions that require sending data samples to the cloud miss the whole point — some of our data cannot be sent to the cloud. 
+For data already in cloud, we can use other cloud-based frameworks.
+Synthesizers are motivated by multiple factors, but in this context, our focus remains on generating synthetic data for on-premise use.
+
+So, what framework did we (initially) choose? Currently we are using the open source version of [SDV](https://sdv.dev/), 
+an easy-to-use framework with a strong community and many useful features out-of-the-box (e.g. built-in evaluators, many modeling techniques). 
+The field of synthetic data is evolving rapidly. While we do not aim to cover the latest advancements exhaustively, the use of foundational models is certainly an area of interest.
+
+**Comments on privacy and privacy preserving techniques**
+
+Ensuring privacy in synthetic data is a non-trivial problem, even if there
+are techniques to ensure levels of privacy, it remains an active area of research. 
+
+*Privacy problems, in synthetic data?*
+
+Yes, when creating synthetic data we build models, models from which we generate synthetic data.
+However, these models can e.g. be overfitted, resulting in leakage of the training data (the real data). 
+Another challenge lies in handling anomalies. If certain information in the real data stands out and we don't account for it when creating synthetic data, there is a risk of backtracking from the synthetic data to the real.
+A typical example could be 'a very rich person in the dataset' or 'a very rare disease'. 
+The problem here, besides from the privacy part, is that it might be the anomalies we really are looking for.
+
+We are experimenting with various differential privacy strategies, but it is still early days, and we do not focus on them in the examples below.
+
+### The Synthetic Data Vault (SDV) - high level
+
+When you initialize and fit a synthesizer (like GaussianCopulaSynthesizer, CTGANSynthesizer, etc.), it trains a model based on 
+the dataset you provide. This model learns the distribution of the data, capturing the relationships and dependencies between 
+different features in the dataset. Below are the (free) synthesizers provided by SDV that we evaluated on each use case.
+
+The synthesizer doesn't memorize individual records from the dataset. 
+Instead, it tries to learn the underlying statistical patterns, correlations, and distributions present in the data. 
+Each synthesizer does this differently:
+
+- **GaussianCopulaSynthesizer:** Models the distribution of each feature using copulas and marginal distributions. 
+- **CTGANSynthesizer:** Uses a GAN (Generative Adversarial Network) approach, which involves training two neural networks (a generator and a discriminator) to model complex data distributions, especially useful for discrete or mixed-type data.
+- **TVAESynthesizer:** Uses a Variational Autoencoder (VAE) approach to learn a lower-dimensional representation of the data and then generate new samples.
+- **CopulaGANSynthesizer:** Combines elements of copulas and GANs to capture complex dependencies.
+- **PARSynthesizer:** Uses Probabilistic AutoRegressive models to model sequential dependencies in the data.
+
+*There are more synthesizers, also from SDV, but not all open source.* We used the first four, when evaluating optimal synthesizer for our different use cases.
+
+**Generators - generating new data - on demand**
+
+Once trained, the synthesizer uses the learned model to generate new synthetic data that follows the same statistical properties and 
+distributions as the original dataset, without directly copying any real data points. If you need more data? Just call the generator.
+
+### Our on-premise analytics platform - ARCUS
+
+<div style="display: flex; align-items: center; gap: 20px;">
+  <p>
+    ARCUS is Telia’s advanced on-premise analytics platform, designed to support a wide range of use cases.
+    The platform provides a Kubeflow-based MLOps environment for descriptive, predictive, generative, and (ongoing) agentic AI. 
+    Fully built on open-source, ARCUS integrates a comprehensive stack of components into a unified platform - where Kubernetes is the cornerstone.
+  </p>
+</div>
+
+### Needed environment to create synthetic data
+
+For an efficient, automated selection of the best synthesizer, we need a number of things - from the underlying platform with GPUs and MLOps (Kubeflow).
+
+- Kubeflow pipelines
+- GPU capabilities (for performance and efficiency)
+- Development (IDE) environment (for framework building and running)
+- Modern data platform (MinIO, Airflow) automating the synthetic data generation datasets
+
+#### Parallelism needed
+
+In the (Kube)flows below, we run evaluations in parallel - one for respective synthesizer, followed by a comparison of usability and fidelity scores, selecting the 'winner'.
+
+*Note:* In earlier version of Kubeflow we noticed that the parallelism wasn't acting as expected, waiting for all threads to complete before moving to next step. We had to create a temporary workaround for this, now solved in Kubeflow Pipelines 2.3.0.
+
+Below, we briefly describe the base flow for selecting synthesizer, followed by one use case where we use the resulting data generator for ML development in cloud.
+
+## Exploring the creation and usefulness of synthetic data
+
+This is what we want to do: We have a use case, the supporting data, and developed ML model.
+
+- How similar is the synthetic data compared to the real data (interesting for e.g. visualization use cases)?
+- How well do the ML models based on synthetic data keep up with ML models based on real data?
+
+**General validation of synthetic data technique**
+
+1. Create synthetic data, save the best synthetic data generator. In this step similarity measures are created by the used (SDV) framework 'out of the box'
+2. Create ML model (in our case classifier model) both on real data, as for synthetic data. Compare performance of both models against the (same) real data testset.
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image-2.png" width="800">
+
+From above, we have an example where the final synthesizer is collected and saved. This step is used in below example, exporting the resulting synthetic data generator to cloud.
+
+## Using synthetic data generators, to enable multiple environments without data transfer
+
+*Below is a usecase where we need to make use of both on-premise and cloud, without moving data to cloud.*
+
+**Problem statement:** 
+
+- Our data cannot be moved from on-premise to cloud.
+- We need extra compute power, in our public cloud environment, to create an ML model for use on-premise.
+- The ML model is to be used on-premise, on new incoming data streams (that cannot be moved to cloud)
+
+**Solution:**
+
+1. Create synthetic data for our on-premise environment use-cases, and - as a side product we save 
+away the synthetic data generator (the pickled model used to create synthetic data). 
+2. Copy the synthetic data generator to cloud
+3. Use the synthetic data generator in cloud, creating synthetic data for training of ML model
+4. Copy ML model to on-premise, and use it for new incoming data
+5. Evaluate: Compare the on-premise AI model with the model created in cloud - against same test data
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image.png" width="700">
+
+
+**Division of work, what is done on-premise Kubeflow, and what is done in cloud (AWS SageMaker)?**
+
+##### On-premise
+
+See above *General validation of synthetic data technique*.
+
+- Develop model on real data – for later comparison with cloud model.
+- Create synthetic generators, evaluate, export best generator to AWS.
+
+##### Cloud
+
+- Use imported (from on-premise) synthetic generator
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: franciscojavierarceo
body: ```suggestion
- Use the imported synthetic generator (from on-premise)
```
created_at: 2025-02-24T16:59:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,190 @@
+---
+title: "AI-Generated Synthetic Data, using Kubeflow pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic data - why and how?
+
+The best results come from real data, but accessing it often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete.
+
+While the above focuses on speed and augmentation, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is it, how do we evaluate?
+
+### Evaluation criteria for synthetic data - how do we know it's useful?
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthesizer is best for our specific dataset and use case.
+
+We need to ensure a good balance between: 
+
+  - Usability (how useful is the synthetic data for your use case), 
+  - Fidelity (how similar), and 
+  - Privacy (approved level for use case)
+
+For now, we are focusing only on usability and fidelity, using framework-provided measurements for fidelity and workflows described below for usability.
+
+### Methods and frameworks creating synthetic data 
+
+For us, open source frameworks are the only ones of interest. 
+Cloud-based solutions that require sending data samples to the cloud miss the whole point — some of our data cannot be sent to the cloud. 
+For data already in cloud, we can use other cloud-based frameworks.
+Synthesizers are motivated by multiple factors, but in this context, our focus remains on generating synthetic data for on-premise use.
+
+So, what framework did we (initially) choose? Currently we are using the open source version of [SDV](https://sdv.dev/), 
+an easy-to-use framework with a strong community and many useful features out-of-the-box (e.g. built-in evaluators, many modeling techniques). 
+The field of synthetic data is evolving rapidly. While we do not aim to cover the latest advancements exhaustively, the use of foundational models is certainly an area of interest.
+
+**Comments on privacy and privacy preserving techniques**
+
+Ensuring privacy in synthetic data is a non-trivial problem, even if there
+are techniques to ensure levels of privacy, it remains an active area of research. 
+
+*Privacy problems, in synthetic data?*
+
+Yes, when creating synthetic data we build models, models from which we generate synthetic data.
+However, these models can e.g. be overfitted, resulting in leakage of the training data (the real data). 
+Another challenge lies in handling anomalies. If certain information in the real data stands out and we don't account for it when creating synthetic data, there is a risk of backtracking from the synthetic data to the real.
+A typical example could be 'a very rich person in the dataset' or 'a very rare disease'. 
+The problem here, besides from the privacy part, is that it might be the anomalies we really are looking for.
+
+We are experimenting with various differential privacy strategies, but it is still early days, and we do not focus on them in the examples below.
+
+### The Synthetic Data Vault (SDV) - high level
+
+When you initialize and fit a synthesizer (like GaussianCopulaSynthesizer, CTGANSynthesizer, etc.), it trains a model based on 
+the dataset you provide. This model learns the distribution of the data, capturing the relationships and dependencies between 
+different features in the dataset. Below are the (free) synthesizers provided by SDV that we evaluated on each use case.
+
+The synthesizer doesn't memorize individual records from the dataset. 
+Instead, it tries to learn the underlying statistical patterns, correlations, and distributions present in the data. 
+Each synthesizer does this differently:
+
+- **GaussianCopulaSynthesizer:** Models the distribution of each feature using copulas and marginal distributions. 
+- **CTGANSynthesizer:** Uses a GAN (Generative Adversarial Network) approach, which involves training two neural networks (a generator and a discriminator) to model complex data distributions, especially useful for discrete or mixed-type data.
+- **TVAESynthesizer:** Uses a Variational Autoencoder (VAE) approach to learn a lower-dimensional representation of the data and then generate new samples.
+- **CopulaGANSynthesizer:** Combines elements of copulas and GANs to capture complex dependencies.
+- **PARSynthesizer:** Uses Probabilistic AutoRegressive models to model sequential dependencies in the data.
+
+*There are more synthesizers, also from SDV, but not all open source.* We used the first four, when evaluating optimal synthesizer for our different use cases.
+
+**Generators - generating new data - on demand**
+
+Once trained, the synthesizer uses the learned model to generate new synthetic data that follows the same statistical properties and 
+distributions as the original dataset, without directly copying any real data points. If you need more data? Just call the generator.
+
+### Our on-premise analytics platform - ARCUS
+
+<div style="display: flex; align-items: center; gap: 20px;">
+  <p>
+    ARCUS is Telia’s advanced on-premise analytics platform, designed to support a wide range of use cases.
+    The platform provides a Kubeflow-based MLOps environment for descriptive, predictive, generative, and (ongoing) agentic AI. 
+    Fully built on open-source, ARCUS integrates a comprehensive stack of components into a unified platform - where Kubernetes is the cornerstone.
+  </p>
+</div>
+
+### Needed environment to create synthetic data
+
+For an efficient, automated selection of the best synthesizer, we need a number of things - from the underlying platform with GPUs and MLOps (Kubeflow).
+
+- Kubeflow pipelines
+- GPU capabilities (for performance and efficiency)
+- Development (IDE) environment (for framework building and running)
+- Modern data platform (MinIO, Airflow) automating the synthetic data generation datasets
+
+#### Parallelism needed
+
+In the (Kube)flows below, we run evaluations in parallel - one for respective synthesizer, followed by a comparison of usability and fidelity scores, selecting the 'winner'.
+
+*Note:* In earlier version of Kubeflow we noticed that the parallelism wasn't acting as expected, waiting for all threads to complete before moving to next step. We had to create a temporary workaround for this, now solved in Kubeflow Pipelines 2.3.0.
+
+Below, we briefly describe the base flow for selecting synthesizer, followed by one use case where we use the resulting data generator for ML development in cloud.
+
+## Exploring the creation and usefulness of synthetic data
+
+This is what we want to do: We have a use case, the supporting data, and developed ML model.
+
+- How similar is the synthetic data compared to the real data (interesting for e.g. visualization use cases)?
+- How well do the ML models based on synthetic data keep up with ML models based on real data?
+
+**General validation of synthetic data technique**
+
+1. Create synthetic data, save the best synthetic data generator. In this step similarity measures are created by the used (SDV) framework 'out of the box'
+2. Create ML model (in our case classifier model) both on real data, as for synthetic data. Compare performance of both models against the (same) real data testset.
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image-2.png" width="800">
+
+From above, we have an example where the final synthesizer is collected and saved. This step is used in below example, exporting the resulting synthetic data generator to cloud.
+
+## Using synthetic data generators, to enable multiple environments without data transfer
+
+*Below is a usecase where we need to make use of both on-premise and cloud, without moving data to cloud.*
+
+**Problem statement:** 
+
+- Our data cannot be moved from on-premise to cloud.
+- We need extra compute power, in our public cloud environment, to create an ML model for use on-premise.
+- The ML model is to be used on-premise, on new incoming data streams (that cannot be moved to cloud)
+
+**Solution:**
+
+1. Create synthetic data for our on-premise environment use-cases, and - as a side product we save 
+away the synthetic data generator (the pickled model used to create synthetic data). 
+2. Copy the synthetic data generator to cloud
+3. Use the synthetic data generator in cloud, creating synthetic data for training of ML model
+4. Copy ML model to on-premise, and use it for new incoming data
+5. Evaluate: Compare the on-premise AI model with the model created in cloud - against same test data
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image.png" width="700">
+
+
+**Division of work, what is done on-premise Kubeflow, and what is done in cloud (AWS SageMaker)?**
+
+##### On-premise
+
+See above *General validation of synthetic data technique*.
+
+- Develop model on real data – for later comparison with cloud model.
+- Create synthetic generators, evaluate, export best generator to AWS.
+
+##### Cloud
+
+- Use imported (from on-premise) synthetic generator
+- Create synthetic data, using synthetic data generator
+- Develop model – compare which synthetic generator is best
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: franciscojavierarceo
body: ```suggestion
- Develop the model and determine which synthetic generator is the best
```
created_at: 2025-02-24T17:01:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,190 @@
+---
+title: "AI-Generated Synthetic Data, using Kubeflow pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic data - why and how?
+
+The best results come from real data, but accessing it often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete.
+
+While the above focuses on speed and augmentation, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is it, how do we evaluate?
+
+### Evaluation criteria for synthetic data - how do we know it's useful?
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthesizer is best for our specific dataset and use case.
+
+We need to ensure a good balance between: 
+
+  - Usability (how useful is the synthetic data for your use case), 
+  - Fidelity (how similar), and 
+  - Privacy (approved level for use case)
+
+For now, we are focusing only on usability and fidelity, using framework-provided measurements for fidelity and workflows described below for usability.
+
+### Methods and frameworks creating synthetic data 
+
+For us, open source frameworks are the only ones of interest. 
+Cloud-based solutions that require sending data samples to the cloud miss the whole point — some of our data cannot be sent to the cloud. 
+For data already in cloud, we can use other cloud-based frameworks.
+Synthesizers are motivated by multiple factors, but in this context, our focus remains on generating synthetic data for on-premise use.
+
+So, what framework did we (initially) choose? Currently we are using the open source version of [SDV](https://sdv.dev/), 
+an easy-to-use framework with a strong community and many useful features out-of-the-box (e.g. built-in evaluators, many modeling techniques). 
+The field of synthetic data is evolving rapidly. While we do not aim to cover the latest advancements exhaustively, the use of foundational models is certainly an area of interest.
+
+**Comments on privacy and privacy preserving techniques**
+
+Ensuring privacy in synthetic data is a non-trivial problem, even if there
+are techniques to ensure levels of privacy, it remains an active area of research. 
+
+*Privacy problems, in synthetic data?*
+
+Yes, when creating synthetic data we build models, models from which we generate synthetic data.
+However, these models can e.g. be overfitted, resulting in leakage of the training data (the real data). 
+Another challenge lies in handling anomalies. If certain information in the real data stands out and we don't account for it when creating synthetic data, there is a risk of backtracking from the synthetic data to the real.
+A typical example could be 'a very rich person in the dataset' or 'a very rare disease'. 
+The problem here, besides from the privacy part, is that it might be the anomalies we really are looking for.
+
+We are experimenting with various differential privacy strategies, but it is still early days, and we do not focus on them in the examples below.
+
+### The Synthetic Data Vault (SDV) - high level
+
+When you initialize and fit a synthesizer (like GaussianCopulaSynthesizer, CTGANSynthesizer, etc.), it trains a model based on 
+the dataset you provide. This model learns the distribution of the data, capturing the relationships and dependencies between 
+different features in the dataset. Below are the (free) synthesizers provided by SDV that we evaluated on each use case.
+
+The synthesizer doesn't memorize individual records from the dataset. 
+Instead, it tries to learn the underlying statistical patterns, correlations, and distributions present in the data. 
+Each synthesizer does this differently:
+
+- **GaussianCopulaSynthesizer:** Models the distribution of each feature using copulas and marginal distributions. 
+- **CTGANSynthesizer:** Uses a GAN (Generative Adversarial Network) approach, which involves training two neural networks (a generator and a discriminator) to model complex data distributions, especially useful for discrete or mixed-type data.
+- **TVAESynthesizer:** Uses a Variational Autoencoder (VAE) approach to learn a lower-dimensional representation of the data and then generate new samples.
+- **CopulaGANSynthesizer:** Combines elements of copulas and GANs to capture complex dependencies.
+- **PARSynthesizer:** Uses Probabilistic AutoRegressive models to model sequential dependencies in the data.
+
+*There are more synthesizers, also from SDV, but not all open source.* We used the first four, when evaluating optimal synthesizer for our different use cases.
+
+**Generators - generating new data - on demand**
+
+Once trained, the synthesizer uses the learned model to generate new synthetic data that follows the same statistical properties and 
+distributions as the original dataset, without directly copying any real data points. If you need more data? Just call the generator.
+
+### Our on-premise analytics platform - ARCUS
+
+<div style="display: flex; align-items: center; gap: 20px;">
+  <p>
+    ARCUS is Telia’s advanced on-premise analytics platform, designed to support a wide range of use cases.
+    The platform provides a Kubeflow-based MLOps environment for descriptive, predictive, generative, and (ongoing) agentic AI. 
+    Fully built on open-source, ARCUS integrates a comprehensive stack of components into a unified platform - where Kubernetes is the cornerstone.
+  </p>
+</div>
+
+### Needed environment to create synthetic data
+
+For an efficient, automated selection of the best synthesizer, we need a number of things - from the underlying platform with GPUs and MLOps (Kubeflow).
+
+- Kubeflow pipelines
+- GPU capabilities (for performance and efficiency)
+- Development (IDE) environment (for framework building and running)
+- Modern data platform (MinIO, Airflow) automating the synthetic data generation datasets
+
+#### Parallelism needed
+
+In the (Kube)flows below, we run evaluations in parallel - one for respective synthesizer, followed by a comparison of usability and fidelity scores, selecting the 'winner'.
+
+*Note:* In earlier version of Kubeflow we noticed that the parallelism wasn't acting as expected, waiting for all threads to complete before moving to next step. We had to create a temporary workaround for this, now solved in Kubeflow Pipelines 2.3.0.
+
+Below, we briefly describe the base flow for selecting synthesizer, followed by one use case where we use the resulting data generator for ML development in cloud.
+
+## Exploring the creation and usefulness of synthetic data
+
+This is what we want to do: We have a use case, the supporting data, and developed ML model.
+
+- How similar is the synthetic data compared to the real data (interesting for e.g. visualization use cases)?
+- How well do the ML models based on synthetic data keep up with ML models based on real data?
+
+**General validation of synthetic data technique**
+
+1. Create synthetic data, save the best synthetic data generator. In this step similarity measures are created by the used (SDV) framework 'out of the box'
+2. Create ML model (in our case classifier model) both on real data, as for synthetic data. Compare performance of both models against the (same) real data testset.
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image-2.png" width="800">
+
+From above, we have an example where the final synthesizer is collected and saved. This step is used in below example, exporting the resulting synthetic data generator to cloud.
+
+## Using synthetic data generators, to enable multiple environments without data transfer
+
+*Below is a usecase where we need to make use of both on-premise and cloud, without moving data to cloud.*
+
+**Problem statement:** 
+
+- Our data cannot be moved from on-premise to cloud.
+- We need extra compute power, in our public cloud environment, to create an ML model for use on-premise.
+- The ML model is to be used on-premise, on new incoming data streams (that cannot be moved to cloud)
+
+**Solution:**
+
+1. Create synthetic data for our on-premise environment use-cases, and - as a side product we save 
+away the synthetic data generator (the pickled model used to create synthetic data). 
+2. Copy the synthetic data generator to cloud
+3. Use the synthetic data generator in cloud, creating synthetic data for training of ML model
+4. Copy ML model to on-premise, and use it for new incoming data
+5. Evaluate: Compare the on-premise AI model with the model created in cloud - against same test data
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image.png" width="700">
+
+
+**Division of work, what is done on-premise Kubeflow, and what is done in cloud (AWS SageMaker)?**
+
+##### On-premise
+
+See above *General validation of synthetic data technique*.
+
+- Develop model on real data – for later comparison with cloud model.
+- Create synthetic generators, evaluate, export best generator to AWS.
+
+##### Cloud
+
+- Use imported (from on-premise) synthetic generator
+- Create synthetic data, using synthetic data generator
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: franciscojavierarceo
body: ```suggestion
- Create synthetic data using the synthetic data generator
```
created_at: 2025-02-24T17:02:08Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,190 @@
+---
+title: "AI-Generated Synthetic Data, using Kubeflow pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic data - why and how?
+
+The best results come from real data, but accessing it often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete.
+
+While the above focuses on speed and augmentation, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is it, how do we evaluate?
+
+### Evaluation criteria for synthetic data - how do we know it's useful?
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthesizer is best for our specific dataset and use case.
+
+We need to ensure a good balance between: 
+
+  - Usability (how useful is the synthetic data for your use case), 
+  - Fidelity (how similar), and 
+  - Privacy (approved level for use case)
+
+For now, we are focusing only on usability and fidelity, using framework-provided measurements for fidelity and workflows described below for usability.
+
+### Methods and frameworks creating synthetic data 
+
+For us, open source frameworks are the only ones of interest. 
+Cloud-based solutions that require sending data samples to the cloud miss the whole point — some of our data cannot be sent to the cloud. 
+For data already in cloud, we can use other cloud-based frameworks.
+Synthesizers are motivated by multiple factors, but in this context, our focus remains on generating synthetic data for on-premise use.
+
+So, what framework did we (initially) choose? Currently we are using the open source version of [SDV](https://sdv.dev/), 
+an easy-to-use framework with a strong community and many useful features out-of-the-box (e.g. built-in evaluators, many modeling techniques). 
+The field of synthetic data is evolving rapidly. While we do not aim to cover the latest advancements exhaustively, the use of foundational models is certainly an area of interest.
+
+**Comments on privacy and privacy preserving techniques**
+
+Ensuring privacy in synthetic data is a non-trivial problem, even if there
+are techniques to ensure levels of privacy, it remains an active area of research. 
+
+*Privacy problems, in synthetic data?*
+
+Yes, when creating synthetic data we build models, models from which we generate synthetic data.
+However, these models can e.g. be overfitted, resulting in leakage of the training data (the real data). 
+Another challenge lies in handling anomalies. If certain information in the real data stands out and we don't account for it when creating synthetic data, there is a risk of backtracking from the synthetic data to the real.
+A typical example could be 'a very rich person in the dataset' or 'a very rare disease'. 
+The problem here, besides from the privacy part, is that it might be the anomalies we really are looking for.
+
+We are experimenting with various differential privacy strategies, but it is still early days, and we do not focus on them in the examples below.
+
+### The Synthetic Data Vault (SDV) - high level
+
+When you initialize and fit a synthesizer (like GaussianCopulaSynthesizer, CTGANSynthesizer, etc.), it trains a model based on 
+the dataset you provide. This model learns the distribution of the data, capturing the relationships and dependencies between 
+different features in the dataset. Below are the (free) synthesizers provided by SDV that we evaluated on each use case.
+
+The synthesizer doesn't memorize individual records from the dataset. 
+Instead, it tries to learn the underlying statistical patterns, correlations, and distributions present in the data. 
+Each synthesizer does this differently:
+
+- **GaussianCopulaSynthesizer:** Models the distribution of each feature using copulas and marginal distributions. 
+- **CTGANSynthesizer:** Uses a GAN (Generative Adversarial Network) approach, which involves training two neural networks (a generator and a discriminator) to model complex data distributions, especially useful for discrete or mixed-type data.
+- **TVAESynthesizer:** Uses a Variational Autoencoder (VAE) approach to learn a lower-dimensional representation of the data and then generate new samples.
+- **CopulaGANSynthesizer:** Combines elements of copulas and GANs to capture complex dependencies.
+- **PARSynthesizer:** Uses Probabilistic AutoRegressive models to model sequential dependencies in the data.
+
+*There are more synthesizers, also from SDV, but not all open source.* We used the first four, when evaluating optimal synthesizer for our different use cases.
+
+**Generators - generating new data - on demand**
+
+Once trained, the synthesizer uses the learned model to generate new synthetic data that follows the same statistical properties and 
+distributions as the original dataset, without directly copying any real data points. If you need more data? Just call the generator.
+
+### Our on-premise analytics platform - ARCUS
+
+<div style="display: flex; align-items: center; gap: 20px;">
+  <p>
+    ARCUS is Telia’s advanced on-premise analytics platform, designed to support a wide range of use cases.
+    The platform provides a Kubeflow-based MLOps environment for descriptive, predictive, generative, and (ongoing) agentic AI. 
+    Fully built on open-source, ARCUS integrates a comprehensive stack of components into a unified platform - where Kubernetes is the cornerstone.
+  </p>
+</div>
+
+### Needed environment to create synthetic data
+
+For an efficient, automated selection of the best synthesizer, we need a number of things - from the underlying platform with GPUs and MLOps (Kubeflow).
+
+- Kubeflow pipelines
+- GPU capabilities (for performance and efficiency)
+- Development (IDE) environment (for framework building and running)
+- Modern data platform (MinIO, Airflow) automating the synthetic data generation datasets
+
+#### Parallelism needed
+
+In the (Kube)flows below, we run evaluations in parallel - one for respective synthesizer, followed by a comparison of usability and fidelity scores, selecting the 'winner'.
+
+*Note:* In earlier version of Kubeflow we noticed that the parallelism wasn't acting as expected, waiting for all threads to complete before moving to next step. We had to create a temporary workaround for this, now solved in Kubeflow Pipelines 2.3.0.
+
+Below, we briefly describe the base flow for selecting synthesizer, followed by one use case where we use the resulting data generator for ML development in cloud.
+
+## Exploring the creation and usefulness of synthetic data
+
+This is what we want to do: We have a use case, the supporting data, and developed ML model.
+
+- How similar is the synthetic data compared to the real data (interesting for e.g. visualization use cases)?
+- How well do the ML models based on synthetic data keep up with ML models based on real data?
+
+**General validation of synthetic data technique**
+
+1. Create synthetic data, save the best synthetic data generator. In this step similarity measures are created by the used (SDV) framework 'out of the box'
+2. Create ML model (in our case classifier model) both on real data, as for synthetic data. Compare performance of both models against the (same) real data testset.
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image-2.png" width="800">
+
+From above, we have an example where the final synthesizer is collected and saved. This step is used in below example, exporting the resulting synthetic data generator to cloud.
+
+## Using synthetic data generators, to enable multiple environments without data transfer
+
+*Below is a usecase where we need to make use of both on-premise and cloud, without moving data to cloud.*
+
+**Problem statement:** 
+
+- Our data cannot be moved from on-premise to cloud.
+- We need extra compute power, in our public cloud environment, to create an ML model for use on-premise.
+- The ML model is to be used on-premise, on new incoming data streams (that cannot be moved to cloud)
+
+**Solution:**
+
+1. Create synthetic data for our on-premise environment use-cases, and - as a side product we save 
+away the synthetic data generator (the pickled model used to create synthetic data). 
+2. Copy the synthetic data generator to cloud
+3. Use the synthetic data generator in cloud, creating synthetic data for training of ML model
+4. Copy ML model to on-premise, and use it for new incoming data
+5. Evaluate: Compare the on-premise AI model with the model created in cloud - against same test data
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image.png" width="700">
+
+
+**Division of work, what is done on-premise Kubeflow, and what is done in cloud (AWS SageMaker)?**
+
+##### On-premise
+
+See above *General validation of synthetic data technique*.
+
+- Develop model on real data – for later comparison with cloud model.
+- Create synthetic generators, evaluate, export best generator to AWS.
+
+##### Cloud
+
+- Use imported (from on-premise) synthetic generator
+- Create synthetic data, using synthetic data generator
+- Develop model – compare which synthetic generator is best
+- Increase synthetic training (if needed) and evaluation data set, to improve model performance
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: franciscojavierarceo
body: I'm not sure what you mean in this sentence.

Do you mean increase the size of the synthetically generated data for training? 
created_at: 2025-02-24T17:02:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,190 @@
+---
+title: "AI-Generated Synthetic Data, using Kubeflow pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic data - why and how?
+
+The best results come from real data, but accessing it often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete.
+
+While the above focuses on speed and augmentation, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is it, how do we evaluate?
+
+### Evaluation criteria for synthetic data - how do we know it's useful?
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthesizer is best for our specific dataset and use case.
+
+We need to ensure a good balance between: 
+
+  - Usability (how useful is the synthetic data for your use case), 
+  - Fidelity (how similar), and 
+  - Privacy (approved level for use case)
+
+For now, we are focusing only on usability and fidelity, using framework-provided measurements for fidelity and workflows described below for usability.
+
+### Methods and frameworks creating synthetic data 
+
+For us, open source frameworks are the only ones of interest. 
+Cloud-based solutions that require sending data samples to the cloud miss the whole point — some of our data cannot be sent to the cloud. 
+For data already in cloud, we can use other cloud-based frameworks.
+Synthesizers are motivated by multiple factors, but in this context, our focus remains on generating synthetic data for on-premise use.
+
+So, what framework did we (initially) choose? Currently we are using the open source version of [SDV](https://sdv.dev/), 
+an easy-to-use framework with a strong community and many useful features out-of-the-box (e.g. built-in evaluators, many modeling techniques). 
+The field of synthetic data is evolving rapidly. While we do not aim to cover the latest advancements exhaustively, the use of foundational models is certainly an area of interest.
+
+**Comments on privacy and privacy preserving techniques**
+
+Ensuring privacy in synthetic data is a non-trivial problem, even if there
+are techniques to ensure levels of privacy, it remains an active area of research. 
+
+*Privacy problems, in synthetic data?*
+
+Yes, when creating synthetic data we build models, models from which we generate synthetic data.
+However, these models can e.g. be overfitted, resulting in leakage of the training data (the real data). 
+Another challenge lies in handling anomalies. If certain information in the real data stands out and we don't account for it when creating synthetic data, there is a risk of backtracking from the synthetic data to the real.
+A typical example could be 'a very rich person in the dataset' or 'a very rare disease'. 
+The problem here, besides from the privacy part, is that it might be the anomalies we really are looking for.
+
+We are experimenting with various differential privacy strategies, but it is still early days, and we do not focus on them in the examples below.
+
+### The Synthetic Data Vault (SDV) - high level
+
+When you initialize and fit a synthesizer (like GaussianCopulaSynthesizer, CTGANSynthesizer, etc.), it trains a model based on 
+the dataset you provide. This model learns the distribution of the data, capturing the relationships and dependencies between 
+different features in the dataset. Below are the (free) synthesizers provided by SDV that we evaluated on each use case.
+
+The synthesizer doesn't memorize individual records from the dataset. 
+Instead, it tries to learn the underlying statistical patterns, correlations, and distributions present in the data. 
+Each synthesizer does this differently:
+
+- **GaussianCopulaSynthesizer:** Models the distribution of each feature using copulas and marginal distributions. 
+- **CTGANSynthesizer:** Uses a GAN (Generative Adversarial Network) approach, which involves training two neural networks (a generator and a discriminator) to model complex data distributions, especially useful for discrete or mixed-type data.
+- **TVAESynthesizer:** Uses a Variational Autoencoder (VAE) approach to learn a lower-dimensional representation of the data and then generate new samples.
+- **CopulaGANSynthesizer:** Combines elements of copulas and GANs to capture complex dependencies.
+- **PARSynthesizer:** Uses Probabilistic AutoRegressive models to model sequential dependencies in the data.
+
+*There are more synthesizers, also from SDV, but not all open source.* We used the first four, when evaluating optimal synthesizer for our different use cases.
+
+**Generators - generating new data - on demand**
+
+Once trained, the synthesizer uses the learned model to generate new synthetic data that follows the same statistical properties and 
+distributions as the original dataset, without directly copying any real data points. If you need more data? Just call the generator.
+
+### Our on-premise analytics platform - ARCUS
+
+<div style="display: flex; align-items: center; gap: 20px;">
+  <p>
+    ARCUS is Telia’s advanced on-premise analytics platform, designed to support a wide range of use cases.
+    The platform provides a Kubeflow-based MLOps environment for descriptive, predictive, generative, and (ongoing) agentic AI. 
+    Fully built on open-source, ARCUS integrates a comprehensive stack of components into a unified platform - where Kubernetes is the cornerstone.
+  </p>
+</div>
+
+### Needed environment to create synthetic data
+
+For an efficient, automated selection of the best synthesizer, we need a number of things - from the underlying platform with GPUs and MLOps (Kubeflow).
+
+- Kubeflow pipelines
+- GPU capabilities (for performance and efficiency)
+- Development (IDE) environment (for framework building and running)
+- Modern data platform (MinIO, Airflow) automating the synthetic data generation datasets
+
+#### Parallelism needed
+
+In the (Kube)flows below, we run evaluations in parallel - one for respective synthesizer, followed by a comparison of usability and fidelity scores, selecting the 'winner'.
+
+*Note:* In earlier version of Kubeflow we noticed that the parallelism wasn't acting as expected, waiting for all threads to complete before moving to next step. We had to create a temporary workaround for this, now solved in Kubeflow Pipelines 2.3.0.
+
+Below, we briefly describe the base flow for selecting synthesizer, followed by one use case where we use the resulting data generator for ML development in cloud.
+
+## Exploring the creation and usefulness of synthetic data
+
+This is what we want to do: We have a use case, the supporting data, and developed ML model.
+
+- How similar is the synthetic data compared to the real data (interesting for e.g. visualization use cases)?
+- How well do the ML models based on synthetic data keep up with ML models based on real data?
+
+**General validation of synthetic data technique**
+
+1. Create synthetic data, save the best synthetic data generator. In this step similarity measures are created by the used (SDV) framework 'out of the box'
+2. Create ML model (in our case classifier model) both on real data, as for synthetic data. Compare performance of both models against the (same) real data testset.
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image-2.png" width="800">
+
+From above, we have an example where the final synthesizer is collected and saved. This step is used in below example, exporting the resulting synthetic data generator to cloud.
+
+## Using synthetic data generators, to enable multiple environments without data transfer
+
+*Below is a usecase where we need to make use of both on-premise and cloud, without moving data to cloud.*
+
+**Problem statement:** 
+
+- Our data cannot be moved from on-premise to cloud.
+- We need extra compute power, in our public cloud environment, to create an ML model for use on-premise.
+- The ML model is to be used on-premise, on new incoming data streams (that cannot be moved to cloud)
+
+**Solution:**
+
+1. Create synthetic data for our on-premise environment use-cases, and - as a side product we save 
+away the synthetic data generator (the pickled model used to create synthetic data). 
+2. Copy the synthetic data generator to cloud
+3. Use the synthetic data generator in cloud, creating synthetic data for training of ML model
+4. Copy ML model to on-premise, and use it for new incoming data
+5. Evaluate: Compare the on-premise AI model with the model created in cloud - against same test data
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image.png" width="700">
+
+
+**Division of work, what is done on-premise Kubeflow, and what is done in cloud (AWS SageMaker)?**
+
+##### On-premise
+
+See above *General validation of synthetic data technique*.
+
+- Develop model on real data – for later comparison with cloud model.
+- Create synthetic generators, evaluate, export best generator to AWS.
+
+##### Cloud
+
+- Use imported (from on-premise) synthetic generator
+- Create synthetic data, using synthetic data generator
+- Develop model – compare which synthetic generator is best
+- Increase synthetic training (if needed) and evaluation data set, to improve model performance
+- Export model to on-premise
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image-3.png" width="500">
+
+
+In some more detail below.
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image-4.png" width="700">
+
+##### On-premise
+
+- Compare real data model against synthetic data model – using real test data.
+
+*In the current examples we see on-par (few percentages lower performance of models created using synthetic data) performance of the ML models. We experimented with increased synthetic dataset, with minor improvements. Augmenting the training data is expected (not tested here) to have more effect when using deep learning algorithms.*
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: franciscojavierarceo
body: ```suggestion
*In the current examples we see near equivalent performance of the ML models (a few percentage points lower for models created using synthetic data). We experimented with increasing the size of the synthetic dataset, with minor improvements. Augmenting the training data is expected (not tested here) to have more effects when using deep learning algorithms.*
```
created_at: 2025-02-24T17:04:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,190 @@
+---
+title: "AI-Generated Synthetic Data, using Kubeflow pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic data - why and how?
+
+The best results come from real data, but accessing it often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete.
+
+While the above focuses on speed and augmentation, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is it, how do we evaluate?
+
+### Evaluation criteria for synthetic data - how do we know it's useful?
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthesizer is best for our specific dataset and use case.
+
+We need to ensure a good balance between: 
+
+  - Usability (how useful is the synthetic data for your use case), 
+  - Fidelity (how similar), and 
+  - Privacy (approved level for use case)
+
+For now, we are focusing only on usability and fidelity, using framework-provided measurements for fidelity and workflows described below for usability.
+
+### Methods and frameworks creating synthetic data 
+
+For us, open source frameworks are the only ones of interest. 
+Cloud-based solutions that require sending data samples to the cloud miss the whole point — some of our data cannot be sent to the cloud. 
+For data already in cloud, we can use other cloud-based frameworks.
+Synthesizers are motivated by multiple factors, but in this context, our focus remains on generating synthetic data for on-premise use.
+
+So, what framework did we (initially) choose? Currently we are using the open source version of [SDV](https://sdv.dev/), 
+an easy-to-use framework with a strong community and many useful features out-of-the-box (e.g. built-in evaluators, many modeling techniques). 
+The field of synthetic data is evolving rapidly. While we do not aim to cover the latest advancements exhaustively, the use of foundational models is certainly an area of interest.
+
+**Comments on privacy and privacy preserving techniques**
+
+Ensuring privacy in synthetic data is a non-trivial problem, even if there
+are techniques to ensure levels of privacy, it remains an active area of research. 
+
+*Privacy problems, in synthetic data?*
+
+Yes, when creating synthetic data we build models, models from which we generate synthetic data.
+However, these models can e.g. be overfitted, resulting in leakage of the training data (the real data). 
+Another challenge lies in handling anomalies. If certain information in the real data stands out and we don't account for it when creating synthetic data, there is a risk of backtracking from the synthetic data to the real.
+A typical example could be 'a very rich person in the dataset' or 'a very rare disease'. 
+The problem here, besides from the privacy part, is that it might be the anomalies we really are looking for.
+
+We are experimenting with various differential privacy strategies, but it is still early days, and we do not focus on them in the examples below.
+
+### The Synthetic Data Vault (SDV) - high level
+
+When you initialize and fit a synthesizer (like GaussianCopulaSynthesizer, CTGANSynthesizer, etc.), it trains a model based on 
+the dataset you provide. This model learns the distribution of the data, capturing the relationships and dependencies between 
+different features in the dataset. Below are the (free) synthesizers provided by SDV that we evaluated on each use case.
+
+The synthesizer doesn't memorize individual records from the dataset. 
+Instead, it tries to learn the underlying statistical patterns, correlations, and distributions present in the data. 
+Each synthesizer does this differently:
+
+- **GaussianCopulaSynthesizer:** Models the distribution of each feature using copulas and marginal distributions. 
+- **CTGANSynthesizer:** Uses a GAN (Generative Adversarial Network) approach, which involves training two neural networks (a generator and a discriminator) to model complex data distributions, especially useful for discrete or mixed-type data.
+- **TVAESynthesizer:** Uses a Variational Autoencoder (VAE) approach to learn a lower-dimensional representation of the data and then generate new samples.
+- **CopulaGANSynthesizer:** Combines elements of copulas and GANs to capture complex dependencies.
+- **PARSynthesizer:** Uses Probabilistic AutoRegressive models to model sequential dependencies in the data.
+
+*There are more synthesizers, also from SDV, but not all open source.* We used the first four, when evaluating optimal synthesizer for our different use cases.
+
+**Generators - generating new data - on demand**
+
+Once trained, the synthesizer uses the learned model to generate new synthetic data that follows the same statistical properties and 
+distributions as the original dataset, without directly copying any real data points. If you need more data? Just call the generator.
+
+### Our on-premise analytics platform - ARCUS
+
+<div style="display: flex; align-items: center; gap: 20px;">
+  <p>
+    ARCUS is Telia’s advanced on-premise analytics platform, designed to support a wide range of use cases.
+    The platform provides a Kubeflow-based MLOps environment for descriptive, predictive, generative, and (ongoing) agentic AI. 
+    Fully built on open-source, ARCUS integrates a comprehensive stack of components into a unified platform - where Kubernetes is the cornerstone.
+  </p>
+</div>
+
+### Needed environment to create synthetic data
+
+For an efficient, automated selection of the best synthesizer, we need a number of things - from the underlying platform with GPUs and MLOps (Kubeflow).
+
+- Kubeflow pipelines
+- GPU capabilities (for performance and efficiency)
+- Development (IDE) environment (for framework building and running)
+- Modern data platform (MinIO, Airflow) automating the synthetic data generation datasets
+
+#### Parallelism needed
+
+In the (Kube)flows below, we run evaluations in parallel - one for respective synthesizer, followed by a comparison of usability and fidelity scores, selecting the 'winner'.
+
+*Note:* In earlier version of Kubeflow we noticed that the parallelism wasn't acting as expected, waiting for all threads to complete before moving to next step. We had to create a temporary workaround for this, now solved in Kubeflow Pipelines 2.3.0.
+
+Below, we briefly describe the base flow for selecting synthesizer, followed by one use case where we use the resulting data generator for ML development in cloud.
+
+## Exploring the creation and usefulness of synthetic data
+
+This is what we want to do: We have a use case, the supporting data, and developed ML model.
+
+- How similar is the synthetic data compared to the real data (interesting for e.g. visualization use cases)?
+- How well do the ML models based on synthetic data keep up with ML models based on real data?
+
+**General validation of synthetic data technique**
+
+1. Create synthetic data, save the best synthetic data generator. In this step similarity measures are created by the used (SDV) framework 'out of the box'
+2. Create ML model (in our case classifier model) both on real data, as for synthetic data. Compare performance of both models against the (same) real data testset.
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image-2.png" width="800">
+
+From above, we have an example where the final synthesizer is collected and saved. This step is used in below example, exporting the resulting synthetic data generator to cloud.
+
+## Using synthetic data generators, to enable multiple environments without data transfer
+
+*Below is a usecase where we need to make use of both on-premise and cloud, without moving data to cloud.*
+
+**Problem statement:** 
+
+- Our data cannot be moved from on-premise to cloud.
+- We need extra compute power, in our public cloud environment, to create an ML model for use on-premise.
+- The ML model is to be used on-premise, on new incoming data streams (that cannot be moved to cloud)
+
+**Solution:**
+
+1. Create synthetic data for our on-premise environment use-cases, and - as a side product we save 
+away the synthetic data generator (the pickled model used to create synthetic data). 
+2. Copy the synthetic data generator to cloud
+3. Use the synthetic data generator in cloud, creating synthetic data for training of ML model
+4. Copy ML model to on-premise, and use it for new incoming data
+5. Evaluate: Compare the on-premise AI model with the model created in cloud - against same test data
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image.png" width="700">
+
+
+**Division of work, what is done on-premise Kubeflow, and what is done in cloud (AWS SageMaker)?**
+
+##### On-premise
+
+See above *General validation of synthetic data technique*.
+
+- Develop model on real data – for later comparison with cloud model.
+- Create synthetic generators, evaluate, export best generator to AWS.
+
+##### Cloud
+
+- Use imported (from on-premise) synthetic generator
+- Create synthetic data, using synthetic data generator
+- Develop model – compare which synthetic generator is best
+- Increase synthetic training (if needed) and evaluation data set, to improve model performance
+- Export model to on-premise
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image-3.png" width="500">
+
+
+In some more detail below.
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image-4.png" width="700">
+
+##### On-premise
+
+- Compare real data model against synthetic data model – using real test data.
+
+*In the current examples we see on-par (few percentages lower performance of models created using synthetic data) performance of the ML models. We experimented with increased synthetic dataset, with minor improvements. Augmenting the training data is expected (not tested here) to have more effect when using deep learning algorithms.*
+
+# Summary, and forward looking
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: franciscojavierarceo
body: ```suggestion
# Summary
```
created_at: 2025-02-24T17:05:09Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,190 @@
+---
+title: "AI-Generated Synthetic Data, using Kubeflow pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic data - why and how?
+
+The best results come from real data, but accessing it often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete.
+
+While the above focuses on speed and augmentation, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is it, how do we evaluate?
+
+### Evaluation criteria for synthetic data - how do we know it's useful?
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthesizer is best for our specific dataset and use case.
+
+We need to ensure a good balance between: 
+
+  - Usability (how useful is the synthetic data for your use case), 
+  - Fidelity (how similar), and 
+  - Privacy (approved level for use case)
+
+For now, we are focusing only on usability and fidelity, using framework-provided measurements for fidelity and workflows described below for usability.
+
+### Methods and frameworks creating synthetic data 
+
+For us, open source frameworks are the only ones of interest. 
+Cloud-based solutions that require sending data samples to the cloud miss the whole point — some of our data cannot be sent to the cloud. 
+For data already in cloud, we can use other cloud-based frameworks.
+Synthesizers are motivated by multiple factors, but in this context, our focus remains on generating synthetic data for on-premise use.
+
+So, what framework did we (initially) choose? Currently we are using the open source version of [SDV](https://sdv.dev/), 
+an easy-to-use framework with a strong community and many useful features out-of-the-box (e.g. built-in evaluators, many modeling techniques). 
+The field of synthetic data is evolving rapidly. While we do not aim to cover the latest advancements exhaustively, the use of foundational models is certainly an area of interest.
+
+**Comments on privacy and privacy preserving techniques**
+
+Ensuring privacy in synthetic data is a non-trivial problem, even if there
+are techniques to ensure levels of privacy, it remains an active area of research. 
+
+*Privacy problems, in synthetic data?*
+
+Yes, when creating synthetic data we build models, models from which we generate synthetic data.
+However, these models can e.g. be overfitted, resulting in leakage of the training data (the real data). 
+Another challenge lies in handling anomalies. If certain information in the real data stands out and we don't account for it when creating synthetic data, there is a risk of backtracking from the synthetic data to the real.
+A typical example could be 'a very rich person in the dataset' or 'a very rare disease'. 
+The problem here, besides from the privacy part, is that it might be the anomalies we really are looking for.
+
+We are experimenting with various differential privacy strategies, but it is still early days, and we do not focus on them in the examples below.
+
+### The Synthetic Data Vault (SDV) - high level
+
+When you initialize and fit a synthesizer (like GaussianCopulaSynthesizer, CTGANSynthesizer, etc.), it trains a model based on 
+the dataset you provide. This model learns the distribution of the data, capturing the relationships and dependencies between 
+different features in the dataset. Below are the (free) synthesizers provided by SDV that we evaluated on each use case.
+
+The synthesizer doesn't memorize individual records from the dataset. 
+Instead, it tries to learn the underlying statistical patterns, correlations, and distributions present in the data. 
+Each synthesizer does this differently:
+
+- **GaussianCopulaSynthesizer:** Models the distribution of each feature using copulas and marginal distributions. 
+- **CTGANSynthesizer:** Uses a GAN (Generative Adversarial Network) approach, which involves training two neural networks (a generator and a discriminator) to model complex data distributions, especially useful for discrete or mixed-type data.
+- **TVAESynthesizer:** Uses a Variational Autoencoder (VAE) approach to learn a lower-dimensional representation of the data and then generate new samples.
+- **CopulaGANSynthesizer:** Combines elements of copulas and GANs to capture complex dependencies.
+- **PARSynthesizer:** Uses Probabilistic AutoRegressive models to model sequential dependencies in the data.
+
+*There are more synthesizers, also from SDV, but not all open source.* We used the first four, when evaluating optimal synthesizer for our different use cases.
+
+**Generators - generating new data - on demand**
+
+Once trained, the synthesizer uses the learned model to generate new synthetic data that follows the same statistical properties and 
+distributions as the original dataset, without directly copying any real data points. If you need more data? Just call the generator.
+
+### Our on-premise analytics platform - ARCUS
+
+<div style="display: flex; align-items: center; gap: 20px;">
+  <p>
+    ARCUS is Telia’s advanced on-premise analytics platform, designed to support a wide range of use cases.
+    The platform provides a Kubeflow-based MLOps environment for descriptive, predictive, generative, and (ongoing) agentic AI. 
+    Fully built on open-source, ARCUS integrates a comprehensive stack of components into a unified platform - where Kubernetes is the cornerstone.
+  </p>
+</div>
+
+### Needed environment to create synthetic data
+
+For an efficient, automated selection of the best synthesizer, we need a number of things - from the underlying platform with GPUs and MLOps (Kubeflow).
+
+- Kubeflow pipelines
+- GPU capabilities (for performance and efficiency)
+- Development (IDE) environment (for framework building and running)
+- Modern data platform (MinIO, Airflow) automating the synthetic data generation datasets
+
+#### Parallelism needed
+
+In the (Kube)flows below, we run evaluations in parallel - one for respective synthesizer, followed by a comparison of usability and fidelity scores, selecting the 'winner'.
+
+*Note:* In earlier version of Kubeflow we noticed that the parallelism wasn't acting as expected, waiting for all threads to complete before moving to next step. We had to create a temporary workaround for this, now solved in Kubeflow Pipelines 2.3.0.
+
+Below, we briefly describe the base flow for selecting synthesizer, followed by one use case where we use the resulting data generator for ML development in cloud.
+
+## Exploring the creation and usefulness of synthetic data
+
+This is what we want to do: We have a use case, the supporting data, and developed ML model.
+
+- How similar is the synthetic data compared to the real data (interesting for e.g. visualization use cases)?
+- How well do the ML models based on synthetic data keep up with ML models based on real data?
+
+**General validation of synthetic data technique**
+
+1. Create synthetic data, save the best synthetic data generator. In this step similarity measures are created by the used (SDV) framework 'out of the box'
+2. Create ML model (in our case classifier model) both on real data, as for synthetic data. Compare performance of both models against the (same) real data testset.
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image-2.png" width="800">
+
+From above, we have an example where the final synthesizer is collected and saved. This step is used in below example, exporting the resulting synthetic data generator to cloud.
+
+## Using synthetic data generators, to enable multiple environments without data transfer
+
+*Below is a usecase where we need to make use of both on-premise and cloud, without moving data to cloud.*
+
+**Problem statement:** 
+
+- Our data cannot be moved from on-premise to cloud.
+- We need extra compute power, in our public cloud environment, to create an ML model for use on-premise.
+- The ML model is to be used on-premise, on new incoming data streams (that cannot be moved to cloud)
+
+**Solution:**
+
+1. Create synthetic data for our on-premise environment use-cases, and - as a side product we save 
+away the synthetic data generator (the pickled model used to create synthetic data). 
+2. Copy the synthetic data generator to cloud
+3. Use the synthetic data generator in cloud, creating synthetic data for training of ML model
+4. Copy ML model to on-premise, and use it for new incoming data
+5. Evaluate: Compare the on-premise AI model with the model created in cloud - against same test data
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image.png" width="700">
+
+
+**Division of work, what is done on-premise Kubeflow, and what is done in cloud (AWS SageMaker)?**
+
+##### On-premise
+
+See above *General validation of synthetic data technique*.
+
+- Develop model on real data – for later comparison with cloud model.
+- Create synthetic generators, evaluate, export best generator to AWS.
+
+##### Cloud
+
+- Use imported (from on-premise) synthetic generator
+- Create synthetic data, using synthetic data generator
+- Develop model – compare which synthetic generator is best
+- Increase synthetic training (if needed) and evaluation data set, to improve model performance
+- Export model to on-premise
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image-3.png" width="500">
+
+
+In some more detail below.
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image-4.png" width="700">
+
+##### On-premise
+
+- Compare real data model against synthetic data model – using real test data.
+
+*In the current examples we see on-par (few percentages lower performance of models created using synthetic data) performance of the ML models. We experimented with increased synthetic dataset, with minor improvements. Augmenting the training data is expected (not tested here) to have more effect when using deep learning algorithms.*
+
+# Summary, and forward looking
+
+Clearly, above flows would be very cumbersome to build and maintain without Kubeflow. 
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: franciscojavierarceo
body: ```suggestion
Clearly, the above workflows would be very cumbersome to build and maintain without Kubeflow. 
```
created_at: 2025-02-24T17:05:29Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,190 @@
+---
+title: "AI-Generated Synthetic Data, using Kubeflow pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic data - why and how?
+
+The best results come from real data, but accessing it often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete.
+
+While the above focuses on speed and augmentation, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is it, how do we evaluate?
+
+### Evaluation criteria for synthetic data - how do we know it's useful?
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthesizer is best for our specific dataset and use case.
+
+We need to ensure a good balance between: 
+
+  - Usability (how useful is the synthetic data for your use case), 
+  - Fidelity (how similar), and 
+  - Privacy (approved level for use case)
+
+For now, we are focusing only on usability and fidelity, using framework-provided measurements for fidelity and workflows described below for usability.
+
+### Methods and frameworks creating synthetic data 
+
+For us, open source frameworks are the only ones of interest. 
+Cloud-based solutions that require sending data samples to the cloud miss the whole point — some of our data cannot be sent to the cloud. 
+For data already in cloud, we can use other cloud-based frameworks.
+Synthesizers are motivated by multiple factors, but in this context, our focus remains on generating synthetic data for on-premise use.
+
+So, what framework did we (initially) choose? Currently we are using the open source version of [SDV](https://sdv.dev/), 
+an easy-to-use framework with a strong community and many useful features out-of-the-box (e.g. built-in evaluators, many modeling techniques). 
+The field of synthetic data is evolving rapidly. While we do not aim to cover the latest advancements exhaustively, the use of foundational models is certainly an area of interest.
+
+**Comments on privacy and privacy preserving techniques**
+
+Ensuring privacy in synthetic data is a non-trivial problem, even if there
+are techniques to ensure levels of privacy, it remains an active area of research. 
+
+*Privacy problems, in synthetic data?*
+
+Yes, when creating synthetic data we build models, models from which we generate synthetic data.
+However, these models can e.g. be overfitted, resulting in leakage of the training data (the real data). 
+Another challenge lies in handling anomalies. If certain information in the real data stands out and we don't account for it when creating synthetic data, there is a risk of backtracking from the synthetic data to the real.
+A typical example could be 'a very rich person in the dataset' or 'a very rare disease'. 
+The problem here, besides from the privacy part, is that it might be the anomalies we really are looking for.
+
+We are experimenting with various differential privacy strategies, but it is still early days, and we do not focus on them in the examples below.
+
+### The Synthetic Data Vault (SDV) - high level
+
+When you initialize and fit a synthesizer (like GaussianCopulaSynthesizer, CTGANSynthesizer, etc.), it trains a model based on 
+the dataset you provide. This model learns the distribution of the data, capturing the relationships and dependencies between 
+different features in the dataset. Below are the (free) synthesizers provided by SDV that we evaluated on each use case.
+
+The synthesizer doesn't memorize individual records from the dataset. 
+Instead, it tries to learn the underlying statistical patterns, correlations, and distributions present in the data. 
+Each synthesizer does this differently:
+
+- **GaussianCopulaSynthesizer:** Models the distribution of each feature using copulas and marginal distributions. 
+- **CTGANSynthesizer:** Uses a GAN (Generative Adversarial Network) approach, which involves training two neural networks (a generator and a discriminator) to model complex data distributions, especially useful for discrete or mixed-type data.
+- **TVAESynthesizer:** Uses a Variational Autoencoder (VAE) approach to learn a lower-dimensional representation of the data and then generate new samples.
+- **CopulaGANSynthesizer:** Combines elements of copulas and GANs to capture complex dependencies.
+- **PARSynthesizer:** Uses Probabilistic AutoRegressive models to model sequential dependencies in the data.
+
+*There are more synthesizers, also from SDV, but not all open source.* We used the first four, when evaluating optimal synthesizer for our different use cases.
+
+**Generators - generating new data - on demand**
+
+Once trained, the synthesizer uses the learned model to generate new synthetic data that follows the same statistical properties and 
+distributions as the original dataset, without directly copying any real data points. If you need more data? Just call the generator.
+
+### Our on-premise analytics platform - ARCUS
+
+<div style="display: flex; align-items: center; gap: 20px;">
+  <p>
+    ARCUS is Telia’s advanced on-premise analytics platform, designed to support a wide range of use cases.
+    The platform provides a Kubeflow-based MLOps environment for descriptive, predictive, generative, and (ongoing) agentic AI. 
+    Fully built on open-source, ARCUS integrates a comprehensive stack of components into a unified platform - where Kubernetes is the cornerstone.
+  </p>
+</div>
+
+### Needed environment to create synthetic data
+
+For an efficient, automated selection of the best synthesizer, we need a number of things - from the underlying platform with GPUs and MLOps (Kubeflow).
+
+- Kubeflow pipelines
+- GPU capabilities (for performance and efficiency)
+- Development (IDE) environment (for framework building and running)
+- Modern data platform (MinIO, Airflow) automating the synthetic data generation datasets
+
+#### Parallelism needed
+
+In the (Kube)flows below, we run evaluations in parallel - one for respective synthesizer, followed by a comparison of usability and fidelity scores, selecting the 'winner'.
+
+*Note:* In earlier version of Kubeflow we noticed that the parallelism wasn't acting as expected, waiting for all threads to complete before moving to next step. We had to create a temporary workaround for this, now solved in Kubeflow Pipelines 2.3.0.
+
+Below, we briefly describe the base flow for selecting synthesizer, followed by one use case where we use the resulting data generator for ML development in cloud.
+
+## Exploring the creation and usefulness of synthetic data
+
+This is what we want to do: We have a use case, the supporting data, and developed ML model.
+
+- How similar is the synthetic data compared to the real data (interesting for e.g. visualization use cases)?
+- How well do the ML models based on synthetic data keep up with ML models based on real data?
+
+**General validation of synthetic data technique**
+
+1. Create synthetic data, save the best synthetic data generator. In this step similarity measures are created by the used (SDV) framework 'out of the box'
+2. Create ML model (in our case classifier model) both on real data, as for synthetic data. Compare performance of both models against the (same) real data testset.
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image-2.png" width="800">
+
+From above, we have an example where the final synthesizer is collected and saved. This step is used in below example, exporting the resulting synthetic data generator to cloud.
+
+## Using synthetic data generators, to enable multiple environments without data transfer
+
+*Below is a usecase where we need to make use of both on-premise and cloud, without moving data to cloud.*
+
+**Problem statement:** 
+
+- Our data cannot be moved from on-premise to cloud.
+- We need extra compute power, in our public cloud environment, to create an ML model for use on-premise.
+- The ML model is to be used on-premise, on new incoming data streams (that cannot be moved to cloud)
+
+**Solution:**
+
+1. Create synthetic data for our on-premise environment use-cases, and - as a side product we save 
+away the synthetic data generator (the pickled model used to create synthetic data). 
+2. Copy the synthetic data generator to cloud
+3. Use the synthetic data generator in cloud, creating synthetic data for training of ML model
+4. Copy ML model to on-premise, and use it for new incoming data
+5. Evaluate: Compare the on-premise AI model with the model created in cloud - against same test data
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image.png" width="700">
+
+
+**Division of work, what is done on-premise Kubeflow, and what is done in cloud (AWS SageMaker)?**
+
+##### On-premise
+
+See above *General validation of synthetic data technique*.
+
+- Develop model on real data – for later comparison with cloud model.
+- Create synthetic generators, evaluate, export best generator to AWS.
+
+##### Cloud
+
+- Use imported (from on-premise) synthetic generator
+- Create synthetic data, using synthetic data generator
+- Develop model – compare which synthetic generator is best
+- Increase synthetic training (if needed) and evaluation data set, to improve model performance
+- Export model to on-premise
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image-3.png" width="500">
+
+
+In some more detail below.
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image-4.png" width="700">
+
+##### On-premise
+
+- Compare real data model against synthetic data model – using real test data.
+
+*In the current examples we see on-par (few percentages lower performance of models created using synthetic data) performance of the ML models. We experimented with increased synthetic dataset, with minor improvements. Augmenting the training data is expected (not tested here) to have more effect when using deep learning algorithms.*
+
+# Summary, and forward looking
+
+Clearly, above flows would be very cumbersome to build and maintain without Kubeflow. 
+All opensource, Kubernetes based platform, Kubeflow, SDV, giving us the needed scalability, robustness, and detailed control.
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: franciscojavierarceo
body: ```suggestion
Our solution is entirely open source, Kubernetes based, and uses Kubeflow and SDV to give us the scalability, robustness, and detailed control that is required.
```
created_at: 2025-02-24T17:06:47Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,190 @@
+---
+title: "AI-Generated Synthetic Data, using Kubeflow pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic data - why and how?
+
+The best results come from real data, but accessing it often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete.
+
+While the above focuses on speed and augmentation, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is it, how do we evaluate?
+
+### Evaluation criteria for synthetic data - how do we know it's useful?
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthesizer is best for our specific dataset and use case.
+
+We need to ensure a good balance between: 
+
+  - Usability (how useful is the synthetic data for your use case), 
+  - Fidelity (how similar), and 
+  - Privacy (approved level for use case)
+
+For now, we are focusing only on usability and fidelity, using framework-provided measurements for fidelity and workflows described below for usability.
+
+### Methods and frameworks creating synthetic data 
+
+For us, open source frameworks are the only ones of interest. 
+Cloud-based solutions that require sending data samples to the cloud miss the whole point — some of our data cannot be sent to the cloud. 
+For data already in cloud, we can use other cloud-based frameworks.
+Synthesizers are motivated by multiple factors, but in this context, our focus remains on generating synthetic data for on-premise use.
+
+So, what framework did we (initially) choose? Currently we are using the open source version of [SDV](https://sdv.dev/), 
+an easy-to-use framework with a strong community and many useful features out-of-the-box (e.g. built-in evaluators, many modeling techniques). 
+The field of synthetic data is evolving rapidly. While we do not aim to cover the latest advancements exhaustively, the use of foundational models is certainly an area of interest.
+
+**Comments on privacy and privacy preserving techniques**
+
+Ensuring privacy in synthetic data is a non-trivial problem, even if there
+are techniques to ensure levels of privacy, it remains an active area of research. 
+
+*Privacy problems, in synthetic data?*
+
+Yes, when creating synthetic data we build models, models from which we generate synthetic data.
+However, these models can e.g. be overfitted, resulting in leakage of the training data (the real data). 
+Another challenge lies in handling anomalies. If certain information in the real data stands out and we don't account for it when creating synthetic data, there is a risk of backtracking from the synthetic data to the real.
+A typical example could be 'a very rich person in the dataset' or 'a very rare disease'. 
+The problem here, besides from the privacy part, is that it might be the anomalies we really are looking for.
+
+We are experimenting with various differential privacy strategies, but it is still early days, and we do not focus on them in the examples below.
+
+### The Synthetic Data Vault (SDV) - high level
+
+When you initialize and fit a synthesizer (like GaussianCopulaSynthesizer, CTGANSynthesizer, etc.), it trains a model based on 
+the dataset you provide. This model learns the distribution of the data, capturing the relationships and dependencies between 
+different features in the dataset. Below are the (free) synthesizers provided by SDV that we evaluated on each use case.
+
+The synthesizer doesn't memorize individual records from the dataset. 
+Instead, it tries to learn the underlying statistical patterns, correlations, and distributions present in the data. 
+Each synthesizer does this differently:
+
+- **GaussianCopulaSynthesizer:** Models the distribution of each feature using copulas and marginal distributions. 
+- **CTGANSynthesizer:** Uses a GAN (Generative Adversarial Network) approach, which involves training two neural networks (a generator and a discriminator) to model complex data distributions, especially useful for discrete or mixed-type data.
+- **TVAESynthesizer:** Uses a Variational Autoencoder (VAE) approach to learn a lower-dimensional representation of the data and then generate new samples.
+- **CopulaGANSynthesizer:** Combines elements of copulas and GANs to capture complex dependencies.
+- **PARSynthesizer:** Uses Probabilistic AutoRegressive models to model sequential dependencies in the data.
+
+*There are more synthesizers, also from SDV, but not all open source.* We used the first four, when evaluating optimal synthesizer for our different use cases.
+
+**Generators - generating new data - on demand**
+
+Once trained, the synthesizer uses the learned model to generate new synthetic data that follows the same statistical properties and 
+distributions as the original dataset, without directly copying any real data points. If you need more data? Just call the generator.
+
+### Our on-premise analytics platform - ARCUS
+
+<div style="display: flex; align-items: center; gap: 20px;">
+  <p>
+    ARCUS is Telia’s advanced on-premise analytics platform, designed to support a wide range of use cases.
+    The platform provides a Kubeflow-based MLOps environment for descriptive, predictive, generative, and (ongoing) agentic AI. 
+    Fully built on open-source, ARCUS integrates a comprehensive stack of components into a unified platform - where Kubernetes is the cornerstone.
+  </p>
+</div>
+
+### Needed environment to create synthetic data
+
+For an efficient, automated selection of the best synthesizer, we need a number of things - from the underlying platform with GPUs and MLOps (Kubeflow).
+
+- Kubeflow pipelines
+- GPU capabilities (for performance and efficiency)
+- Development (IDE) environment (for framework building and running)
+- Modern data platform (MinIO, Airflow) automating the synthetic data generation datasets
+
+#### Parallelism needed
+
+In the (Kube)flows below, we run evaluations in parallel - one for respective synthesizer, followed by a comparison of usability and fidelity scores, selecting the 'winner'.
+
+*Note:* In earlier version of Kubeflow we noticed that the parallelism wasn't acting as expected, waiting for all threads to complete before moving to next step. We had to create a temporary workaround for this, now solved in Kubeflow Pipelines 2.3.0.
+
+Below, we briefly describe the base flow for selecting synthesizer, followed by one use case where we use the resulting data generator for ML development in cloud.
+
+## Exploring the creation and usefulness of synthetic data
+
+This is what we want to do: We have a use case, the supporting data, and developed ML model.
+
+- How similar is the synthetic data compared to the real data (interesting for e.g. visualization use cases)?
+- How well do the ML models based on synthetic data keep up with ML models based on real data?
+
+**General validation of synthetic data technique**
+
+1. Create synthetic data, save the best synthetic data generator. In this step similarity measures are created by the used (SDV) framework 'out of the box'
+2. Create ML model (in our case classifier model) both on real data, as for synthetic data. Compare performance of both models against the (same) real data testset.
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image-2.png" width="800">
+
+From above, we have an example where the final synthesizer is collected and saved. This step is used in below example, exporting the resulting synthetic data generator to cloud.
+
+## Using synthetic data generators, to enable multiple environments without data transfer
+
+*Below is a usecase where we need to make use of both on-premise and cloud, without moving data to cloud.*
+
+**Problem statement:** 
+
+- Our data cannot be moved from on-premise to cloud.
+- We need extra compute power, in our public cloud environment, to create an ML model for use on-premise.
+- The ML model is to be used on-premise, on new incoming data streams (that cannot be moved to cloud)
+
+**Solution:**
+
+1. Create synthetic data for our on-premise environment use-cases, and - as a side product we save 
+away the synthetic data generator (the pickled model used to create synthetic data). 
+2. Copy the synthetic data generator to cloud
+3. Use the synthetic data generator in cloud, creating synthetic data for training of ML model
+4. Copy ML model to on-premise, and use it for new incoming data
+5. Evaluate: Compare the on-premise AI model with the model created in cloud - against same test data
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image.png" width="700">
+
+
+**Division of work, what is done on-premise Kubeflow, and what is done in cloud (AWS SageMaker)?**
+
+##### On-premise
+
+See above *General validation of synthetic data technique*.
+
+- Develop model on real data – for later comparison with cloud model.
+- Create synthetic generators, evaluate, export best generator to AWS.
+
+##### Cloud
+
+- Use imported (from on-premise) synthetic generator
+- Create synthetic data, using synthetic data generator
+- Develop model – compare which synthetic generator is best
+- Increase synthetic training (if needed) and evaluation data set, to improve model performance
+- Export model to on-premise
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image-3.png" width="500">
+
+
+In some more detail below.
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image-4.png" width="700">
+
+##### On-premise
+
+- Compare real data model against synthetic data model – using real test data.
+
+*In the current examples we see on-par (few percentages lower performance of models created using synthetic data) performance of the ML models. We experimented with increased synthetic dataset, with minor improvements. Augmenting the training data is expected (not tested here) to have more effect when using deep learning algorithms.*
+
+# Summary, and forward looking
+
+Clearly, above flows would be very cumbersome to build and maintain without Kubeflow. 
+All opensource, Kubernetes based platform, Kubeflow, SDV, giving us the needed scalability, robustness, and detailed control.
+
+The area of synthetic data generation is moving fast with the overall AI field. 
+Reports of increasing use of synthetic data for e.g. LLM training is frequent, but the application areas are many more.
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: franciscojavierarceo
body: ```suggestion
Reports of the increased usage of synthetic data for e.g. LLM training is frequent but the application areas are much greater.
```
created_at: 2025-02-24T17:07:19Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,190 @@
+---
+title: "AI-Generated Synthetic Data, using Kubeflow pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic data - why and how?
+
+The best results come from real data, but accessing it often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete.
+
+While the above focuses on speed and augmentation, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is it, how do we evaluate?
+
+### Evaluation criteria for synthetic data - how do we know it's useful?
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthesizer is best for our specific dataset and use case.
+
+We need to ensure a good balance between: 
+
+  - Usability (how useful is the synthetic data for your use case), 
+  - Fidelity (how similar), and 
+  - Privacy (approved level for use case)
+
+For now, we are focusing only on usability and fidelity, using framework-provided measurements for fidelity and workflows described below for usability.
+
+### Methods and frameworks creating synthetic data 
+
+For us, open source frameworks are the only ones of interest. 
+Cloud-based solutions that require sending data samples to the cloud miss the whole point — some of our data cannot be sent to the cloud. 
+For data already in cloud, we can use other cloud-based frameworks.
+Synthesizers are motivated by multiple factors, but in this context, our focus remains on generating synthetic data for on-premise use.
+
+So, what framework did we (initially) choose? Currently we are using the open source version of [SDV](https://sdv.dev/), 
+an easy-to-use framework with a strong community and many useful features out-of-the-box (e.g. built-in evaluators, many modeling techniques). 
+The field of synthetic data is evolving rapidly. While we do not aim to cover the latest advancements exhaustively, the use of foundational models is certainly an area of interest.
+
+**Comments on privacy and privacy preserving techniques**
+
+Ensuring privacy in synthetic data is a non-trivial problem, even if there
+are techniques to ensure levels of privacy, it remains an active area of research. 
+
+*Privacy problems, in synthetic data?*
+
+Yes, when creating synthetic data we build models, models from which we generate synthetic data.
+However, these models can e.g. be overfitted, resulting in leakage of the training data (the real data). 
+Another challenge lies in handling anomalies. If certain information in the real data stands out and we don't account for it when creating synthetic data, there is a risk of backtracking from the synthetic data to the real.
+A typical example could be 'a very rich person in the dataset' or 'a very rare disease'. 
+The problem here, besides from the privacy part, is that it might be the anomalies we really are looking for.
+
+We are experimenting with various differential privacy strategies, but it is still early days, and we do not focus on them in the examples below.
+
+### The Synthetic Data Vault (SDV) - high level
+
+When you initialize and fit a synthesizer (like GaussianCopulaSynthesizer, CTGANSynthesizer, etc.), it trains a model based on 
+the dataset you provide. This model learns the distribution of the data, capturing the relationships and dependencies between 
+different features in the dataset. Below are the (free) synthesizers provided by SDV that we evaluated on each use case.
+
+The synthesizer doesn't memorize individual records from the dataset. 
+Instead, it tries to learn the underlying statistical patterns, correlations, and distributions present in the data. 
+Each synthesizer does this differently:
+
+- **GaussianCopulaSynthesizer:** Models the distribution of each feature using copulas and marginal distributions. 
+- **CTGANSynthesizer:** Uses a GAN (Generative Adversarial Network) approach, which involves training two neural networks (a generator and a discriminator) to model complex data distributions, especially useful for discrete or mixed-type data.
+- **TVAESynthesizer:** Uses a Variational Autoencoder (VAE) approach to learn a lower-dimensional representation of the data and then generate new samples.
+- **CopulaGANSynthesizer:** Combines elements of copulas and GANs to capture complex dependencies.
+- **PARSynthesizer:** Uses Probabilistic AutoRegressive models to model sequential dependencies in the data.
+
+*There are more synthesizers, also from SDV, but not all open source.* We used the first four, when evaluating optimal synthesizer for our different use cases.
+
+**Generators - generating new data - on demand**
+
+Once trained, the synthesizer uses the learned model to generate new synthetic data that follows the same statistical properties and 
+distributions as the original dataset, without directly copying any real data points. If you need more data? Just call the generator.
+
+### Our on-premise analytics platform - ARCUS
+
+<div style="display: flex; align-items: center; gap: 20px;">
+  <p>
+    ARCUS is Telia’s advanced on-premise analytics platform, designed to support a wide range of use cases.
+    The platform provides a Kubeflow-based MLOps environment for descriptive, predictive, generative, and (ongoing) agentic AI. 
+    Fully built on open-source, ARCUS integrates a comprehensive stack of components into a unified platform - where Kubernetes is the cornerstone.
+  </p>
+</div>
+
+### Needed environment to create synthetic data
+
+For an efficient, automated selection of the best synthesizer, we need a number of things - from the underlying platform with GPUs and MLOps (Kubeflow).
+
+- Kubeflow pipelines
+- GPU capabilities (for performance and efficiency)
+- Development (IDE) environment (for framework building and running)
+- Modern data platform (MinIO, Airflow) automating the synthetic data generation datasets
+
+#### Parallelism needed
+
+In the (Kube)flows below, we run evaluations in parallel - one for respective synthesizer, followed by a comparison of usability and fidelity scores, selecting the 'winner'.
+
+*Note:* In earlier version of Kubeflow we noticed that the parallelism wasn't acting as expected, waiting for all threads to complete before moving to next step. We had to create a temporary workaround for this, now solved in Kubeflow Pipelines 2.3.0.
+
+Below, we briefly describe the base flow for selecting synthesizer, followed by one use case where we use the resulting data generator for ML development in cloud.
+
+## Exploring the creation and usefulness of synthetic data
+
+This is what we want to do: We have a use case, the supporting data, and developed ML model.
+
+- How similar is the synthetic data compared to the real data (interesting for e.g. visualization use cases)?
+- How well do the ML models based on synthetic data keep up with ML models based on real data?
+
+**General validation of synthetic data technique**
+
+1. Create synthetic data, save the best synthetic data generator. In this step similarity measures are created by the used (SDV) framework 'out of the box'
+2. Create ML model (in our case classifier model) both on real data, as for synthetic data. Compare performance of both models against the (same) real data testset.
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image-2.png" width="800">
+
+From above, we have an example where the final synthesizer is collected and saved. This step is used in below example, exporting the resulting synthetic data generator to cloud.
+
+## Using synthetic data generators, to enable multiple environments without data transfer
+
+*Below is a usecase where we need to make use of both on-premise and cloud, without moving data to cloud.*
+
+**Problem statement:** 
+
+- Our data cannot be moved from on-premise to cloud.
+- We need extra compute power, in our public cloud environment, to create an ML model for use on-premise.
+- The ML model is to be used on-premise, on new incoming data streams (that cannot be moved to cloud)
+
+**Solution:**
+
+1. Create synthetic data for our on-premise environment use-cases, and - as a side product we save 
+away the synthetic data generator (the pickled model used to create synthetic data). 
+2. Copy the synthetic data generator to cloud
+3. Use the synthetic data generator in cloud, creating synthetic data for training of ML model
+4. Copy ML model to on-premise, and use it for new incoming data
+5. Evaluate: Compare the on-premise AI model with the model created in cloud - against same test data
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image.png" width="700">
+
+
+**Division of work, what is done on-premise Kubeflow, and what is done in cloud (AWS SageMaker)?**
+
+##### On-premise
+
+See above *General validation of synthetic data technique*.
+
+- Develop model on real data – for later comparison with cloud model.
+- Create synthetic generators, evaluate, export best generator to AWS.
+
+##### Cloud
+
+- Use imported (from on-premise) synthetic generator
+- Create synthetic data, using synthetic data generator
+- Develop model – compare which synthetic generator is best
+- Increase synthetic training (if needed) and evaluation data set, to improve model performance
+- Export model to on-premise
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image-3.png" width="500">
+
+
+In some more detail below.
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image-4.png" width="700">
+
+##### On-premise
+
+- Compare real data model against synthetic data model – using real test data.
+
+*In the current examples we see on-par (few percentages lower performance of models created using synthetic data) performance of the ML models. We experimented with increased synthetic dataset, with minor improvements. Augmenting the training data is expected (not tested here) to have more effect when using deep learning algorithms.*
+
+# Summary, and forward looking
+
+Clearly, above flows would be very cumbersome to build and maintain without Kubeflow. 
+All opensource, Kubernetes based platform, Kubeflow, SDV, giving us the needed scalability, robustness, and detailed control.
+
+The area of synthetic data generation is moving fast with the overall AI field. 
+Reports of increasing use of synthetic data for e.g. LLM training is frequent, but the application areas are many more.
+We also expect more capable synthesizers and, hopefully, privacy preserving techniques keeping up with these. 
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: franciscojavierarceo
body: ```suggestion
We also expect more capable synthesizers and, hopefully, privacy preserving techniques to keep up with the innovation in this area.
```
created_at: 2025-02-24T17:08:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,190 @@
+---
+title: "AI-Generated Synthetic Data, using Kubeflow pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic data - why and how?
+
+The best results come from real data, but accessing it often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete.
+
+While the above focuses on speed and augmentation, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is it, how do we evaluate?
+
+### Evaluation criteria for synthetic data - how do we know it's useful?
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthesizer is best for our specific dataset and use case.
+
+We need to ensure a good balance between: 
+
+  - Usability (how useful is the synthetic data for your use case), 
+  - Fidelity (how similar), and 
+  - Privacy (approved level for use case)
+
+For now, we are focusing only on usability and fidelity, using framework-provided measurements for fidelity and workflows described below for usability.
+
+### Methods and frameworks creating synthetic data 
+
+For us, open source frameworks are the only ones of interest. 
+Cloud-based solutions that require sending data samples to the cloud miss the whole point — some of our data cannot be sent to the cloud. 
+For data already in cloud, we can use other cloud-based frameworks.
+Synthesizers are motivated by multiple factors, but in this context, our focus remains on generating synthetic data for on-premise use.
+
+So, what framework did we (initially) choose? Currently we are using the open source version of [SDV](https://sdv.dev/), 
+an easy-to-use framework with a strong community and many useful features out-of-the-box (e.g. built-in evaluators, many modeling techniques). 
+The field of synthetic data is evolving rapidly. While we do not aim to cover the latest advancements exhaustively, the use of foundational models is certainly an area of interest.
+
+**Comments on privacy and privacy preserving techniques**
+
+Ensuring privacy in synthetic data is a non-trivial problem, even if there
+are techniques to ensure levels of privacy, it remains an active area of research. 
+
+*Privacy problems, in synthetic data?*
+
+Yes, when creating synthetic data we build models, models from which we generate synthetic data.
+However, these models can e.g. be overfitted, resulting in leakage of the training data (the real data). 
+Another challenge lies in handling anomalies. If certain information in the real data stands out and we don't account for it when creating synthetic data, there is a risk of backtracking from the synthetic data to the real.
+A typical example could be 'a very rich person in the dataset' or 'a very rare disease'. 
+The problem here, besides from the privacy part, is that it might be the anomalies we really are looking for.
+
+We are experimenting with various differential privacy strategies, but it is still early days, and we do not focus on them in the examples below.
+
+### The Synthetic Data Vault (SDV) - high level
+
+When you initialize and fit a synthesizer (like GaussianCopulaSynthesizer, CTGANSynthesizer, etc.), it trains a model based on 
+the dataset you provide. This model learns the distribution of the data, capturing the relationships and dependencies between 
+different features in the dataset. Below are the (free) synthesizers provided by SDV that we evaluated on each use case.
+
+The synthesizer doesn't memorize individual records from the dataset. 
+Instead, it tries to learn the underlying statistical patterns, correlations, and distributions present in the data. 
+Each synthesizer does this differently:
+
+- **GaussianCopulaSynthesizer:** Models the distribution of each feature using copulas and marginal distributions. 
+- **CTGANSynthesizer:** Uses a GAN (Generative Adversarial Network) approach, which involves training two neural networks (a generator and a discriminator) to model complex data distributions, especially useful for discrete or mixed-type data.
+- **TVAESynthesizer:** Uses a Variational Autoencoder (VAE) approach to learn a lower-dimensional representation of the data and then generate new samples.
+- **CopulaGANSynthesizer:** Combines elements of copulas and GANs to capture complex dependencies.
+- **PARSynthesizer:** Uses Probabilistic AutoRegressive models to model sequential dependencies in the data.
+
+*There are more synthesizers, also from SDV, but not all open source.* We used the first four, when evaluating optimal synthesizer for our different use cases.
+
+**Generators - generating new data - on demand**
+
+Once trained, the synthesizer uses the learned model to generate new synthetic data that follows the same statistical properties and 
+distributions as the original dataset, without directly copying any real data points. If you need more data? Just call the generator.
+
+### Our on-premise analytics platform - ARCUS
+
+<div style="display: flex; align-items: center; gap: 20px;">
+  <p>
+    ARCUS is Telia’s advanced on-premise analytics platform, designed to support a wide range of use cases.
+    The platform provides a Kubeflow-based MLOps environment for descriptive, predictive, generative, and (ongoing) agentic AI. 
+    Fully built on open-source, ARCUS integrates a comprehensive stack of components into a unified platform - where Kubernetes is the cornerstone.
+  </p>
+</div>
+
+### Needed environment to create synthetic data
+
+For an efficient, automated selection of the best synthesizer, we need a number of things - from the underlying platform with GPUs and MLOps (Kubeflow).
+
+- Kubeflow pipelines
+- GPU capabilities (for performance and efficiency)
+- Development (IDE) environment (for framework building and running)
+- Modern data platform (MinIO, Airflow) automating the synthetic data generation datasets
+
+#### Parallelism needed
+
+In the (Kube)flows below, we run evaluations in parallel - one for respective synthesizer, followed by a comparison of usability and fidelity scores, selecting the 'winner'.
+
+*Note:* In earlier version of Kubeflow we noticed that the parallelism wasn't acting as expected, waiting for all threads to complete before moving to next step. We had to create a temporary workaround for this, now solved in Kubeflow Pipelines 2.3.0.
+
+Below, we briefly describe the base flow for selecting synthesizer, followed by one use case where we use the resulting data generator for ML development in cloud.
+
+## Exploring the creation and usefulness of synthetic data
+
+This is what we want to do: We have a use case, the supporting data, and developed ML model.
+
+- How similar is the synthetic data compared to the real data (interesting for e.g. visualization use cases)?
+- How well do the ML models based on synthetic data keep up with ML models based on real data?
+
+**General validation of synthetic data technique**
+
+1. Create synthetic data, save the best synthetic data generator. In this step similarity measures are created by the used (SDV) framework 'out of the box'
+2. Create ML model (in our case classifier model) both on real data, as for synthetic data. Compare performance of both models against the (same) real data testset.
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image-2.png" width="800">
+
+From above, we have an example where the final synthesizer is collected and saved. This step is used in below example, exporting the resulting synthetic data generator to cloud.
+
+## Using synthetic data generators, to enable multiple environments without data transfer
+
+*Below is a usecase where we need to make use of both on-premise and cloud, without moving data to cloud.*
+
+**Problem statement:** 
+
+- Our data cannot be moved from on-premise to cloud.
+- We need extra compute power, in our public cloud environment, to create an ML model for use on-premise.
+- The ML model is to be used on-premise, on new incoming data streams (that cannot be moved to cloud)
+
+**Solution:**
+
+1. Create synthetic data for our on-premise environment use-cases, and - as a side product we save 
+away the synthetic data generator (the pickled model used to create synthetic data). 
+2. Copy the synthetic data generator to cloud
+3. Use the synthetic data generator in cloud, creating synthetic data for training of ML model
+4. Copy ML model to on-premise, and use it for new incoming data
+5. Evaluate: Compare the on-premise AI model with the model created in cloud - against same test data
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image.png" width="700">
+
+
+**Division of work, what is done on-premise Kubeflow, and what is done in cloud (AWS SageMaker)?**
+
+##### On-premise
+
+See above *General validation of synthetic data technique*.
+
+- Develop model on real data – for later comparison with cloud model.
+- Create synthetic generators, evaluate, export best generator to AWS.
+
+##### Cloud
+
+- Use imported (from on-premise) synthetic generator
+- Create synthetic data, using synthetic data generator
+- Develop model – compare which synthetic generator is best
+- Increase synthetic training (if needed) and evaluation data set, to improve model performance
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: akeed
body: Increasing the amount of synthetic data needed to improve the model performance. So, more synthetic training data, and more synthetic evaluation data - not more test data, that is still real data, waiting on-premise for comparison with the model developed using 'only' real data. 
created_at: 2025-02-24T19:14:44Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,190 @@
+---
+title: "AI-Generated Synthetic Data, using Kubeflow pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic data - why and how?
+
+The best results come from real data, but accessing it often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete.
+
+While the above focuses on speed and augmentation, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is it, how do we evaluate?
+
+### Evaluation criteria for synthetic data - how do we know it's useful?
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthesizer is best for our specific dataset and use case.
+
+We need to ensure a good balance between: 
+
+  - Usability (how useful is the synthetic data for your use case), 
+  - Fidelity (how similar), and 
+  - Privacy (approved level for use case)
+
+For now, we are focusing only on usability and fidelity, using framework-provided measurements for fidelity and workflows described below for usability.
+
+### Methods and frameworks creating synthetic data 
+
+For us, open source frameworks are the only ones of interest. 
+Cloud-based solutions that require sending data samples to the cloud miss the whole point — some of our data cannot be sent to the cloud. 
+For data already in cloud, we can use other cloud-based frameworks.
+Synthesizers are motivated by multiple factors, but in this context, our focus remains on generating synthetic data for on-premise use.
+
+So, what framework did we (initially) choose? Currently we are using the open source version of [SDV](https://sdv.dev/), 
+an easy-to-use framework with a strong community and many useful features out-of-the-box (e.g. built-in evaluators, many modeling techniques). 
+The field of synthetic data is evolving rapidly. While we do not aim to cover the latest advancements exhaustively, the use of foundational models is certainly an area of interest.
+
+**Comments on privacy and privacy preserving techniques**
+
+Ensuring privacy in synthetic data is a non-trivial problem, even if there
+are techniques to ensure levels of privacy, it remains an active area of research. 
+
+*Privacy problems, in synthetic data?*
+
+Yes, when creating synthetic data we build models, models from which we generate synthetic data.
+However, these models can e.g. be overfitted, resulting in leakage of the training data (the real data). 
+Another challenge lies in handling anomalies. If certain information in the real data stands out and we don't account for it when creating synthetic data, there is a risk of backtracking from the synthetic data to the real.
+A typical example could be 'a very rich person in the dataset' or 'a very rare disease'. 
+The problem here, besides from the privacy part, is that it might be the anomalies we really are looking for.
+
+We are experimenting with various differential privacy strategies, but it is still early days, and we do not focus on them in the examples below.
+
+### The Synthetic Data Vault (SDV) - high level
+
+When you initialize and fit a synthesizer (like GaussianCopulaSynthesizer, CTGANSynthesizer, etc.), it trains a model based on 
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: akeed
body: They appear directly on the landing page of (link above, the https://sdv.dev), and not sure if they (SDV) might move around their sub-links (not the sdv.dev, I hope).
created_at: 2025-02-24T19:19:17Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,190 @@
+---
+title: "AI-Generated Synthetic Data, using Kubeflow pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic data - why and how?
+
+The best results come from real data, but accessing it often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete.
+
+While the above focuses on speed and augmentation, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is it, how do we evaluate?
+
+### Evaluation criteria for synthetic data - how do we know it's useful?
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthesizer is best for our specific dataset and use case.
+
+We need to ensure a good balance between: 
+
+  - Usability (how useful is the synthetic data for your use case), 
+  - Fidelity (how similar), and 
+  - Privacy (approved level for use case)
+
+For now, we are focusing only on usability and fidelity, using framework-provided measurements for fidelity and workflows described below for usability.
+
+### Methods and frameworks creating synthetic data 
+
+For us, open source frameworks are the only ones of interest. 
+Cloud-based solutions that require sending data samples to the cloud miss the whole point — some of our data cannot be sent to the cloud. 
+For data already in cloud, we can use other cloud-based frameworks.
+Synthesizers are motivated by multiple factors, but in this context, our focus remains on generating synthetic data for on-premise use.
+
+So, what framework did we (initially) choose? Currently we are using the open source version of [SDV](https://sdv.dev/), 
+an easy-to-use framework with a strong community and many useful features out-of-the-box (e.g. built-in evaluators, many modeling techniques). 
+The field of synthetic data is evolving rapidly. While we do not aim to cover the latest advancements exhaustively, the use of foundational models is certainly an area of interest.
+
+**Comments on privacy and privacy preserving techniques**
+
+Ensuring privacy in synthetic data is a non-trivial problem, even if there
+are techniques to ensure levels of privacy, it remains an active area of research. 
+
+*Privacy problems, in synthetic data?*
+
+Yes, when creating synthetic data we build models, models from which we generate synthetic data.
+However, these models can e.g. be overfitted, resulting in leakage of the training data (the real data). 
+Another challenge lies in handling anomalies. If certain information in the real data stands out and we don't account for it when creating synthetic data, there is a risk of backtracking from the synthetic data to the real.
+A typical example could be 'a very rich person in the dataset' or 'a very rare disease'. 
+The problem here, besides from the privacy part, is that it might be the anomalies we really are looking for.
+
+We are experimenting with various differential privacy strategies, but it is still early days, and we do not focus on them in the examples below.
+
+### The Synthetic Data Vault (SDV) - high level
+
+When you initialize and fit a synthesizer (like GaussianCopulaSynthesizer, CTGANSynthesizer, etc.), it trains a model based on 
+the dataset you provide. This model learns the distribution of the data, capturing the relationships and dependencies between 
+different features in the dataset. Below are the (free) synthesizers provided by SDV that we evaluated on each use case.
+
+The synthesizer doesn't memorize individual records from the dataset. 
+Instead, it tries to learn the underlying statistical patterns, correlations, and distributions present in the data. 
+Each synthesizer does this differently:
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: akeed
body: Thanks :-)
created_at: 2025-02-24T19:42:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,190 @@
+---
+title: "AI-Generated Synthetic Data, using Kubeflow pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic data - why and how?
+
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: akeed
body: Updated
created_at: 2025-02-24T19:44:10Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,190 @@
+---
+title: "AI-Generated Synthetic Data, using Kubeflow pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic data - why and how?
+
+The best results come from real data, but accessing it often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete.
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: akeed
body: Updated
created_at: 2025-02-24T19:44:13Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,190 @@
+---
+title: "AI-Generated Synthetic Data, using Kubeflow pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic data - why and how?
+
+The best results come from real data, but accessing it often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete.
+
+While the above focuses on speed and augmentation, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: akeed
body: Thanks, updated.
created_at: 2025-02-24T19:44:15Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,190 @@
+---
+title: "AI-Generated Synthetic Data, using Kubeflow pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic data - why and how?
+
+The best results come from real data, but accessing it often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete.
+
+While the above focuses on speed and augmentation, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is it, how do we evaluate?
+
+### Evaluation criteria for synthetic data - how do we know it's useful?
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthesizer is best for our specific dataset and use case.
+
+We need to ensure a good balance between: 
+
+  - Usability (how useful is the synthetic data for your use case), 
+  - Fidelity (how similar), and 
+  - Privacy (approved level for use case)
+
+For now, we are focusing only on usability and fidelity, using framework-provided measurements for fidelity and workflows described below for usability.
+
+### Methods and frameworks creating synthetic data 
+
+For us, open source frameworks are the only ones of interest. 
+Cloud-based solutions that require sending data samples to the cloud miss the whole point — some of our data cannot be sent to the cloud. 
+For data already in cloud, we can use other cloud-based frameworks.
+Synthesizers are motivated by multiple factors, but in this context, our focus remains on generating synthetic data for on-premise use.
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: akeed
body: Thanks, added.
created_at: 2025-02-24T19:44:21Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,190 @@
+---
+title: "Synthetic Data Generation with Kubeflow Pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic Data Generation - Why and How?
+
+When creating insights, decisions, and actions from data, the best results come from real data. But accessing real data often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete. Synthetic data, using different techniques (see below) generating data that mimics the statistical properties of real data, opens up a good number of use cases. Synthetic data, depening on technique used, not only have similar statistical base properties as the real data, but can also preserve correlations between features in the data. 
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: franciscojavierarceo
body: ```suggestion
When creating insights, decisions, and actions from data, the best results come from real data. But accessing real data often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete. 

Synthetic data, uses different techniques outlined below to generate data that mimics the statistical properties of real data, which opens up a good number of use cases. Synthetic data, depending on technique used, not only has similar statistical base properties as the real data, but also preserves correlations between features in the data. 
```
created_at: 2025-02-26T15:40:16Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,190 @@
+---
+title: "AI-Generated Synthetic Data, using Kubeflow pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic data - why and how?
+
+The best results come from real data, but accessing it often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete.
+
+While the above focuses on speed and augmentation, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is it, how do we evaluate?
+
+### Evaluation criteria for synthetic data - how do we know it's useful?
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthesizer is best for our specific dataset and use case.
+
+We need to ensure a good balance between: 
+
+  - Usability (how useful is the synthetic data for your use case), 
+  - Fidelity (how similar), and 
+  - Privacy (approved level for use case)
+
+For now, we are focusing only on usability and fidelity, using framework-provided measurements for fidelity and workflows described below for usability.
+
+### Methods and frameworks creating synthetic data 
+
+For us, open source frameworks are the only ones of interest. 
+Cloud-based solutions that require sending data samples to the cloud miss the whole point — some of our data cannot be sent to the cloud. 
+For data already in cloud, we can use other cloud-based frameworks.
+Synthesizers are motivated by multiple factors, but in this context, our focus remains on generating synthetic data for on-premise use.
+
+So, what framework did we (initially) choose? Currently we are using the open source version of [SDV](https://sdv.dev/), 
+an easy-to-use framework with a strong community and many useful features out-of-the-box (e.g. built-in evaluators, many modeling techniques). 
+The field of synthetic data is evolving rapidly. While we do not aim to cover the latest advancements exhaustively, the use of foundational models is certainly an area of interest.
+
+**Comments on privacy and privacy preserving techniques**
+
+Ensuring privacy in synthetic data is a non-trivial problem, even if there
+are techniques to ensure levels of privacy, it remains an active area of research. 
+
+*Privacy problems, in synthetic data?*
+
+Yes, when creating synthetic data we build models, models from which we generate synthetic data.
+However, these models can e.g. be overfitted, resulting in leakage of the training data (the real data). 
+Another challenge lies in handling anomalies. If certain information in the real data stands out and we don't account for it when creating synthetic data, there is a risk of backtracking from the synthetic data to the real.
+A typical example could be 'a very rich person in the dataset' or 'a very rare disease'. 
+The problem here, besides from the privacy part, is that it might be the anomalies we really are looking for.
+
+We are experimenting with various differential privacy strategies, but it is still early days, and we do not focus on them in the examples below.
+
+### The Synthetic Data Vault (SDV) - high level
+
+When you initialize and fit a synthesizer (like GaussianCopulaSynthesizer, CTGANSynthesizer, etc.), it trains a model based on 
+the dataset you provide. This model learns the distribution of the data, capturing the relationships and dependencies between 
+different features in the dataset. Below are the (free) synthesizers provided by SDV that we evaluated on each use case.
+
+The synthesizer doesn't memorize individual records from the dataset. 
+Instead, it tries to learn the underlying statistical patterns, correlations, and distributions present in the data. 
+Each synthesizer does this differently:
+
+- **GaussianCopulaSynthesizer:** Models the distribution of each feature using copulas and marginal distributions. 
+- **CTGANSynthesizer:** Uses a GAN (Generative Adversarial Network) approach, which involves training two neural networks (a generator and a discriminator) to model complex data distributions, especially useful for discrete or mixed-type data.
+- **TVAESynthesizer:** Uses a Variational Autoencoder (VAE) approach to learn a lower-dimensional representation of the data and then generate new samples.
+- **CopulaGANSynthesizer:** Combines elements of copulas and GANs to capture complex dependencies.
+- **PARSynthesizer:** Uses Probabilistic AutoRegressive models to model sequential dependencies in the data.
+
+*There are more synthesizers, also from SDV, but not all open source.* We used the first four, when evaluating optimal synthesizer for our different use cases.
+
+**Generators - generating new data - on demand**
+
+Once trained, the synthesizer uses the learned model to generate new synthetic data that follows the same statistical properties and 
+distributions as the original dataset, without directly copying any real data points. If you need more data? Just call the generator.
+
+### Our on-premise analytics platform - ARCUS
+
+<div style="display: flex; align-items: center; gap: 20px;">
+  <p>
+    ARCUS is Telia’s advanced on-premise analytics platform, designed to support a wide range of use cases.
+    The platform provides a Kubeflow-based MLOps environment for descriptive, predictive, generative, and (ongoing) agentic AI. 
+    Fully built on open-source, ARCUS integrates a comprehensive stack of components into a unified platform - where Kubernetes is the cornerstone.
+  </p>
+</div>
+
+### Needed environment to create synthetic data
+
+For an efficient, automated selection of the best synthesizer, we need a number of things - from the underlying platform with GPUs and MLOps (Kubeflow).
+
+- Kubeflow pipelines
+- GPU capabilities (for performance and efficiency)
+- Development (IDE) environment (for framework building and running)
+- Modern data platform (MinIO, Airflow) automating the synthetic data generation datasets
+
+#### Parallelism needed
+
+In the (Kube)flows below, we run evaluations in parallel - one for respective synthesizer, followed by a comparison of usability and fidelity scores, selecting the 'winner'.
+
+*Note:* In earlier version of Kubeflow we noticed that the parallelism wasn't acting as expected, waiting for all threads to complete before moving to next step. We had to create a temporary workaround for this, now solved in Kubeflow Pipelines 2.3.0.
+
+Below, we briefly describe the base flow for selecting synthesizer, followed by one use case where we use the resulting data generator for ML development in cloud.
+
+## Exploring the creation and usefulness of synthetic data
+
+This is what we want to do: We have a use case, the supporting data, and developed ML model.
+
+- How similar is the synthetic data compared to the real data (interesting for e.g. visualization use cases)?
+- How well do the ML models based on synthetic data keep up with ML models based on real data?
+
+**General validation of synthetic data technique**
+
+1. Create synthetic data, save the best synthetic data generator. In this step similarity measures are created by the used (SDV) framework 'out of the box'
+2. Create ML model (in our case classifier model) both on real data, as for synthetic data. Compare performance of both models against the (same) real data testset.
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image-2.png" width="800">
+
+From above, we have an example where the final synthesizer is collected and saved. This step is used in below example, exporting the resulting synthetic data generator to cloud.
+
+## Using synthetic data generators, to enable multiple environments without data transfer
+
+*Below is a usecase where we need to make use of both on-premise and cloud, without moving data to cloud.*
+
+**Problem statement:** 
+
+- Our data cannot be moved from on-premise to cloud.
+- We need extra compute power, in our public cloud environment, to create an ML model for use on-premise.
+- The ML model is to be used on-premise, on new incoming data streams (that cannot be moved to cloud)
+
+**Solution:**
+
+1. Create synthetic data for our on-premise environment use-cases, and - as a side product we save 
+away the synthetic data generator (the pickled model used to create synthetic data). 
+2. Copy the synthetic data generator to cloud
+3. Use the synthetic data generator in cloud, creating synthetic data for training of ML model
+4. Copy ML model to on-premise, and use it for new incoming data
+5. Evaluate: Compare the on-premise AI model with the model created in cloud - against same test data
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image.png" width="700">
+
+
+**Division of work, what is done on-premise Kubeflow, and what is done in cloud (AWS SageMaker)?**
+
+##### On-premise
+
+See above *General validation of synthetic data technique*.
+
+- Develop model on real data – for later comparison with cloud model.
+- Create synthetic generators, evaluate, export best generator to AWS.
+
+##### Cloud
+
+- Use imported (from on-premise) synthetic generator
+- Create synthetic data, using synthetic data generator
+- Develop model – compare which synthetic generator is best
+- Increase synthetic training (if needed) and evaluation data set, to improve model performance
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: akeed
body: Updated to below:

- Use the imported synthetic generator (from on-premise)
- Create synthetic data using the synthetic data generator
- Develop the model and determine which synthetic generator is the best
- Increase the amount of synthetic data, to see if the increase of synthetic data improves model performance (not for sure it will, see below comment)

created_at: 2025-02-26T16:14:15Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,192 @@
+---
+title: "Synthetic Data Generation with Kubeflow Pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic Data Generation - Why and How?
+
+When creating insights, decisions, and actions from data, the best results come from real data. But accessing real data often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete. 
+
+Synthetic data, uses different techniques outlined below to generate data that mimics the statistical properties of real data, which opens up a good number of use cases. Synthetic data, depending on technique used, not only has similar statistical base properties as the real data, but also preserves correlations between features in the data. 
+
+While the above focuses on speed of development in general, and augmentation of data to improve performance of analytical modes, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is synthetic data, how do we evaluate it?
+
+### Evaluation Criteria for Synthetic Data
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthetic data generator technique (synthesizer) is best for our specific dataset and use case. More on synthesizers below, when looking into the framework (SDV) we selected. For now, synthetizers are statistical and (more often) AI models trained to mimic the real data. Once developed, the resulting models are used to create as much synthetic data as you find useful for your use case. 
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: rimolive
body: Shouldn't it be `synthetic data generation` instead of `synthetic data generator`?
created_at: 2025-02-26T16:54:19Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,192 @@
+---
+title: "Synthetic Data Generation with Kubeflow Pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic Data Generation - Why and How?
+
+When creating insights, decisions, and actions from data, the best results come from real data. But accessing real data often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete. 
+
+Synthetic data, uses different techniques outlined below to generate data that mimics the statistical properties of real data, which opens up a good number of use cases. Synthetic data, depending on technique used, not only has similar statistical base properties as the real data, but also preserves correlations between features in the data. 
+
+While the above focuses on speed of development in general, and augmentation of data to improve performance of analytical modes, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is synthetic data, how do we evaluate it?
+
+### Evaluation Criteria for Synthetic Data
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthetic data generator technique (synthesizer) is best for our specific dataset and use case. More on synthesizers below, when looking into the framework (SDV) we selected. For now, synthetizers are statistical and (more often) AI models trained to mimic the real data. Once developed, the resulting models are used to create as much synthetic data as you find useful for your use case. 
+
+We need to ensure a good balance between: 
+
+  - Usability (how useful is the synthetic data for your use case), 
+  - Fidelity (how similar), and 
+  - Privacy (approved level for use case)
+
+For now, we are focusing only on usability and fidelity, using framework-provided measurements for fidelity and workflows described below for usability.
+
+### Frameworks for Creating Synthetic Data 
+
+This post focuses exclusively on open source frameworks.
+Some data cannot be sent to the cloud, so some cloud-based SDG solutions are not always a good fit. 
+For data already in cloud, we can use other cloud-based frameworks.
+Synthesizers are motivated by multiple factors, but in this context, our focus remains on generating synthetic data for on-premise use.
+
+So, what framework did we (initially) choose? Currently we are using the open source version of [SDV](https://sdv.dev/), 
+an easy-to-use framework with a strong community and many useful features out-of-the-box (e.g. built-in evaluators, many modeling techniques). 
+The field of synthetic data is evolving rapidly. While we do not aim to cover the latest advancements exhaustively, the use of Foundation models is certainly an area of interest.
+
+**Comments on privacy and privacy preserving techniques**
+
+Ensuring privacy in synthetic data is a non-trivial problem, even if there
+are techniques to ensure levels of privacy, it remains an active area of research. 
+
+*Privacy problems, in synthetic data?*
+
+When creating synthetic data we build models, models from which we generate synthetic data, which can expose privacy problems.
+However, these models can be overfit, resulting in leakage of the training data (i.e., the real data). 
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: rimolive
body: I don't know if I followed this. Is this a limitation of SDG or the problem relies only on training?
created_at: 2025-02-26T17:01:27Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,192 @@
+---
+title: "Synthetic Data Generation with Kubeflow Pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic Data Generation - Why and How?
+
+When creating insights, decisions, and actions from data, the best results come from real data. But accessing real data often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete. 
+
+Synthetic data, uses different techniques outlined below to generate data that mimics the statistical properties of real data, which opens up a good number of use cases. Synthetic data, depending on technique used, not only has similar statistical base properties as the real data, but also preserves correlations between features in the data. 
+
+While the above focuses on speed of development in general, and augmentation of data to improve performance of analytical modes, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is synthetic data, how do we evaluate it?
+
+### Evaluation Criteria for Synthetic Data
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthetic data generator technique (synthesizer) is best for our specific dataset and use case. More on synthesizers below, when looking into the framework (SDV) we selected. For now, synthetizers are statistical and (more often) AI models trained to mimic the real data. Once developed, the resulting models are used to create as much synthetic data as you find useful for your use case. 
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: akeed
body: You're right. Both generation and generator is used, but here generation is the correct one. (Generator is the (cool) pkl model that you can call - and it will generate more synthetic data as you wish). Updated, thanks
created_at: 2025-02-26T21:18:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,192 @@
+---
+title: "Synthetic Data Generation with Kubeflow Pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic Data Generation - Why and How?
+
+When creating insights, decisions, and actions from data, the best results come from real data. But accessing real data often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete. 
+
+Synthetic data, uses different techniques outlined below to generate data that mimics the statistical properties of real data, which opens up a good number of use cases. Synthetic data, depending on technique used, not only has similar statistical base properties as the real data, but also preserves correlations between features in the data. 
+
+While the above focuses on speed of development in general, and augmentation of data to improve performance of analytical modes, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is synthetic data, how do we evaluate it?
+
+### Evaluation Criteria for Synthetic Data
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthetic data generator technique (synthesizer) is best for our specific dataset and use case. More on synthesizers below, when looking into the framework (SDV) we selected. For now, synthetizers are statistical and (more often) AI models trained to mimic the real data. Once developed, the resulting models are used to create as much synthetic data as you find useful for your use case. 
+
+We need to ensure a good balance between: 
+
+  - Usability (how useful is the synthetic data for your use case), 
+  - Fidelity (how similar), and 
+  - Privacy (approved level for use case)
+
+For now, we are focusing only on usability and fidelity, using framework-provided measurements for fidelity and workflows described below for usability.
+
+### Frameworks for Creating Synthetic Data 
+
+This post focuses exclusively on open source frameworks.
+Some data cannot be sent to the cloud, so some cloud-based SDG solutions are not always a good fit. 
+For data already in cloud, we can use other cloud-based frameworks.
+Synthesizers are motivated by multiple factors, but in this context, our focus remains on generating synthetic data for on-premise use.
+
+So, what framework did we (initially) choose? Currently we are using the open source version of [SDV](https://sdv.dev/), 
+an easy-to-use framework with a strong community and many useful features out-of-the-box (e.g. built-in evaluators, many modeling techniques). 
+The field of synthetic data is evolving rapidly. While we do not aim to cover the latest advancements exhaustively, the use of Foundation models is certainly an area of interest.
+
+**Comments on privacy and privacy preserving techniques**
+
+Ensuring privacy in synthetic data is a non-trivial problem, even if there
+are techniques to ensure levels of privacy, it remains an active area of research. 
+
+*Privacy problems, in synthetic data?*
+
+When creating synthetic data we build models, models from which we generate synthetic data, which can expose privacy problems.
+However, these models can be overfit, resulting in leakage of the training data (i.e., the real data). 
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: akeed
body: It's the problem of models memorizing what it was trained on (the real data). The goal is to create similar data, not mirroring real data. It's more complicated than that, but I only hinted about the challenges with preserving privacy. 

Updated to below. Not perfect English, even I can see that...

When creating synthetic data we build models, models from which we generate synthetic data, which can expose privacy problems.
These models can e.g. be overfitted (close, or fully, memorizing the training data), resulting in leakage of the real data it was trained on. 
created_at: 2025-02-26T21:25:07Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,192 @@
+---
+title: "Synthetic Data Generation with Kubeflow Pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic Data Generation - Why and How?
+
+When creating insights, decisions, and actions from data, the best results come from real data. But accessing real data often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete. 
+
+Synthetic data, uses different techniques outlined below to generate data that mimics the statistical properties of real data, which opens up a good number of use cases. Synthetic data, depending on technique used, not only has similar statistical base properties as the real data, but also preserves correlations between features in the data. 
+
+While the above focuses on speed of development in general, and augmentation of data to improve performance of analytical modes, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is synthetic data, how do we evaluate it?
+
+### Evaluation Criteria for Synthetic Data
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthetic data generation technique (synthesizer) is best for our specific dataset and use case. More on synthesizers below, when looking into the framework (SDV) we selected. For now, synthetizers are statistical and (more often) AI models trained to mimic the real data. Once developed, the resulting models are used to create as much synthetic data as you find useful for your use case. 
+
+We need to ensure a good balance between: 
+
+  - Usability (how useful is the synthetic data for your use case), 
+  - Fidelity (how similar), and 
+  - Privacy (approved level for use case)
+
+For now, we are focusing only on usability and fidelity, using framework-provided measurements for fidelity and workflows described below for usability.
+
+### Frameworks for Creating Synthetic Data 
+
+This post focuses exclusively on open source frameworks.
+Some data cannot be sent to the cloud, so some cloud-based SDG solutions are not always a good fit. 
+For data already in cloud, we can use other cloud-based frameworks.
+Synthesizers are motivated by multiple factors, but in this context, our focus remains on generating synthetic data for on-premise use.
+
+So, what framework did we (initially) choose? Currently we are using the open source version of [SDV](https://sdv.dev/), 
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: hbelmiro
body: ```suggestion
So, what framework did we (initially) choose? Currently, we are using the open source version of [SDV](https://sdv.dev/), 
```
created_at: 2025-02-27T11:10:09Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,192 @@
+---
+title: "Synthetic Data Generation with Kubeflow Pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic Data Generation - Why and How?
+
+When creating insights, decisions, and actions from data, the best results come from real data. But accessing real data often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete. 
+
+Synthetic data, uses different techniques outlined below to generate data that mimics the statistical properties of real data, which opens up a good number of use cases. Synthetic data, depending on technique used, not only has similar statistical base properties as the real data, but also preserves correlations between features in the data. 
+
+While the above focuses on speed of development in general, and augmentation of data to improve performance of analytical modes, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is synthetic data, how do we evaluate it?
+
+### Evaluation Criteria for Synthetic Data
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthetic data generation technique (synthesizer) is best for our specific dataset and use case. More on synthesizers below, when looking into the framework (SDV) we selected. For now, synthetizers are statistical and (more often) AI models trained to mimic the real data. Once developed, the resulting models are used to create as much synthetic data as you find useful for your use case. 
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: hbelmiro
body: ```suggestion
There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthetic data generation technique (synthesizer) is best for our specific dataset and use case. More on synthesizers below, when looking into the framework (SDV) we selected. For now, synthesizers are statistical and (more often) AI models trained to mimic the real data. Once developed, the resulting models are used to create as much synthetic data as you find useful for your use case. 
```
created_at: 2025-02-27T11:14:34Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,192 @@
+---
+title: "Synthetic Data Generation with Kubeflow Pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic Data Generation - Why and How?
+
+When creating insights, decisions, and actions from data, the best results come from real data. But accessing real data often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete. 
+
+Synthetic data, uses different techniques outlined below to generate data that mimics the statistical properties of real data, which opens up a good number of use cases. Synthetic data, depending on technique used, not only has similar statistical base properties as the real data, but also preserves correlations between features in the data. 
+
+While the above focuses on speed of development in general, and augmentation of data to improve performance of analytical modes, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is synthetic data, how do we evaluate it?
+
+### Evaluation Criteria for Synthetic Data
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthetic data generation technique (synthesizer) is best for our specific dataset and use case. More on synthesizers below, when looking into the framework (SDV) we selected. For now, synthesizers are statistical and (more often) AI models trained to mimic the real data. Once developed, the resulting models are used to create as much synthetic data as you find useful for your use case. 
+
+We need to ensure a good balance between: 
+
+  - Usability (how useful is the synthetic data for your use case), 
+  - Fidelity (how similar), and 
+  - Privacy (approved level for use case)
+
+For now, we are focusing only on usability and fidelity, using framework-provided measurements for fidelity and workflows described below for usability.
+
+### Frameworks for Creating Synthetic Data 
+
+This post focuses exclusively on open source frameworks.
+Some data cannot be sent to the cloud, so some cloud-based SDG solutions are not always a good fit. 
+For data already in cloud, we can use other cloud-based frameworks.
+Synthesizers are motivated by multiple factors, but in this context, our focus remains on generating synthetic data for on-premise use.
+
+So, what framework did we (initially) choose? Currently, we are using the open source version of [SDV](https://sdv.dev/), 
+an easy-to-use framework with a strong community and many useful features out-of-the-box (e.g. built-in evaluators, many modeling techniques). 
+The field of synthetic data is evolving rapidly. While we do not aim to cover the latest advancements exhaustively, the use of Foundation models is certainly an area of interest.
+
+**Comments on privacy and privacy preserving techniques**
+
+Ensuring privacy in synthetic data is a non-trivial problem, even if there
+are techniques to ensure levels of privacy, it remains an active area of research. 
+
+*Privacy problems, in synthetic data?*
+
+When creating synthetic data we build models, models from which we generate synthetic data, which can expose privacy problems.
+These models can e.g. be overfitted (close, or fully, memorizing the training data), resulting in leakage of the real data it was trained on. 
+Another challenge lies in handling anomalies. If certain information in the real data stands out and we don't account for it when creating synthetic data, there is a risk of backtracking from the synthetic data to the real.
+A typical example could be 'a very rich person in the dataset' or 'a very rare disease'. 
+The problem here, besides from the privacy part, is that it might be the anomalies we really are looking for.
+
+We are experimenting with various differential privacy strategies, but it is still early days, and we do not focus on them in the examples below.
+
+### The Synthetic Data Vault (SDV)
+
+When you initialize and fit a synthesizer (like GaussianCopulaSynthesizer, CTGANSynthesizer, etc.), it trains a model based on 
+the dataset you provide. This model learns the distribution of the data, capturing the relationships and dependencies between 
+different features in the dataset. Below are the (free) synthesizers provided by SDV that we evaluated on each use case.
+
+The synthesizer doesn't memorize individual records from the dataset. 
+Instead, it tries to learn the underlying statistical patterns, correlations, and distributions present in the data. 
+Each synthesizer does this differently:
+
+- **GaussianCopulaSynthesizer:** Models the distribution of each feature using copulas and marginal distributions. 
+- **CTGANSynthesizer:** Uses a GAN (Generative Adversarial Network) approach, which involves training two neural networks (a generator and a discriminator) to model complex data distributions, especially useful for discrete or mixed-type data.
+- **TVAESynthesizer:** Uses a Variational Autoencoder (VAE) approach to learn a lower-dimensional representation of the data and then generate new samples.
+- **CopulaGANSynthesizer:** Combines elements of copulas and GANs to capture complex dependencies.
+- **PARSynthesizer:** Uses Probabilistic AutoRegressive models to model sequential dependencies in the data.
+
+*There are more synthesizers, also from SDV, but not all are open source.* We used the first four, when evaluating optimal synthesizer for our different use cases.
+
+**Generators - generating new data - on demand**
+
+Once trained, the synthesizer uses the learned model to generate new synthetic data that follows the same statistical properties and 
+distributions as the original dataset, without directly copying any real data points. If you need more data? Just call the generator.
+
+### Our On-Premise Analytics Platform: ARCUS
+
+<div style="display: flex; align-items: center; gap: 20px;">
+  <p>
+    ARCUS is Telia’s advanced on-premise analytics platform, designed to support a wide range of use cases.
+    The platform provides a Kubeflow-based MLOps environment for descriptive, predictive, generative, and (ongoing) agentic AI. 
+    Fully built on open-source, ARCUS integrates a comprehensive stack of components into a unified platform - where Kubernetes is the cornerstone.
+  </p>
+</div>
+
+### Needed environment to create synthetic data
+
+For an efficient, automated selection of the best synthesizer, we need a number of things - from the underlying platform with GPUs and MLOps (Kubeflow).
+
+- Kubeflow pipelines
+- GPU capabilities (for performance and efficiency)
+- Development (IDE) environment (for framework building and running)
+- Modern data platform (MinIO, Airflow) automating the synthetic data generation datasets
+
+#### Parallelism needed
+
+In the (Kube)flows below, we run evaluations in parallel - one for respective synthesizer, followed by a comparison of usability and fidelity scores, selecting the 'winner'.
+
+*Note:* In earlier version of Kubeflow we noticed that the parallelism wasn't acting as expected, waiting for all threads to complete before moving to next step. We had to create a temporary workaround for this, now solved in Kubeflow Pipelines 2.3.0.
+
+Below, we briefly describe the base flow for selecting synthesizer, followed by one use case where we use the resulting data generator for ML development in cloud.
+
+## Exploring the Creation and Usefulness of Synthetic Data
+
+This is what we want to do: we have a use case, the supporting data, and developed ML model.
+
+- How similar is the synthetic data compared to the real data (interesting for e.g. visualization use cases)?
+- How well do the ML models based on synthetic data keep up with ML models based on real data?
+
+**Validation of synthetic data techniques**
+
+1. Create the synthetic data and save the best synthetic data generator. In this step similarity measures are created by the out of the box SDV framework
+2. Create the ML model (in our case classifier model) both on real data and the using the synthetic data. Compare the performance of both models against the same real data testset.
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image-2.png" width="800">
+
+From above, we have an example where the final synthesizer is collected and saved. This step is used in the example  below, exporting the resulting synthetic data generator to cloud.
+
+## Using Synthetic Data Generators to Enable Multiple Environments without Data Transfer
+
+*Below is a usecase where we need to make use of both on-premise and cloud, without moving data to cloud.*
+
+**Problem statement:** 
+
+- Our data cannot be moved from on-premise to cloud.
+- We need extra compute power, in our public cloud environment, to create an ML model for use on-premise.
+- The ML model is to be used on-premise, on new incoming data streams (that cannot be moved to cloud)
+
+**Solution:**
+
+1. Create synthetic data for our on-premise environment use-cases, and - as a side product we save 
+away the synthetic data generator (the pickled model used to create synthetic data). 
+2. Copy the synthetic data generator to cloud
+3. Use the synthetic data generator in the cloud, creating synthetic data for training of an ML model
+4. Copy the ML model on-premise, and use it for new incoming data
+5. Evaluate: Compare the on-premise AI model with the model created in the cloud - against the same test data
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image.png" width="700">
+
+
+**Division of work, what is done on-premise with Kubeflow, and what is done in cloud (AWS SageMaker)?**
+
+##### On-premise
+
+See the above *Validation of synthetic data techniques*.
+
+- Develop the model on real data – for the comparison later with the cloud model.
+- Create synthetic generators, evaluate the generators, and export the best generator to AWS.
+
+##### Cloud
+
+- Use the imported synthetic generator (from on-premise)
+- Create synthetic data using the synthetic data generator
+- Develop the model and determine which synthetic generator is the best
+- Increase the amount of synthetic data, to see if the increase of synthetic data improves model performance (not for sure it will, see below comment)
+- Export model to on-premise
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image-3.png" width="500">
+
+
+In some more detail below.
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image-4.png" width="700">
+
+##### On-premise
+
+- Compare real data model against synthetic data model – using real test data.
+
+*In the current examples we see near equivalent performance of the ML models (a few percentage points lower for models created using synthetic data). We experimented with increasing the size of the synthetic dataset, with minor improvements. Augmenting the training data is expected (not tested here) to have more effects when using deep learning algorithms.*
+
+# Summary
+
+Clearly, the above workflows would be very cumbersome to build and maintain without Kubeflow. 
+Our solution is entirely open source, Kubernetes based, and uses Kubeflow and SDV to give us the scalability, robustness, and detailed control that is required.
+
+The area of synthetic data generation is moving fast with the overall AI field. 
+Reports of the increased usage of synthetic data for e.g. LLM training is frequent but the application areas are much greater.
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: varodrig
body: can we cite a reference here? to connect with our reference to reports.

created_at: 2025-02-27T23:41:42Z
author_association: NONE
diff_hunk: @@ -0,0 +1,192 @@
+---
+title: "Synthetic Data Generation with Kubeflow Pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic Data Generation - Why and How?
+
+When creating insights, decisions, and actions from data, the best results come from real data. But accessing real data often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete. 
+
+Synthetic data, uses different techniques outlined below to generate data that mimics the statistical properties of real data, which opens up a good number of use cases. Synthetic data, depending on technique used, not only has similar statistical base properties as the real data, but also preserves correlations between features in the data. 
+
+While the above focuses on speed of development in general, and augmentation of data to improve performance of analytical modes, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: varodrig
body: suggestion: to expand a little the bullet points, for example to  explain why we think it will bring a better privacy and security. 
created_at: 2025-02-27T23:43:40Z
author_association: NONE
diff_hunk: @@ -0,0 +1,192 @@
+---
+title: "Synthetic Data Generation with Kubeflow Pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic Data Generation - Why and How?
+
+When creating insights, decisions, and actions from data, the best results come from real data. But accessing real data often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete. 
+
+Synthetic data, uses different techniques outlined below to generate data that mimics the statistical properties of real data, which opens up a good number of use cases. Synthetic data, depending on technique used, not only has similar statistical base properties as the real data, but also preserves correlations between features in the data. 
+
+While the above focuses on speed of development in general, and augmentation of data to improve performance of analytical modes, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is synthetic data, how do we evaluate it?
+
+### Evaluation Criteria for Synthetic Data
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthetic data generation technique (synthesizer) is best for our specific dataset and use case. More on synthesizers below, when looking into the framework (SDV) we selected. For now, synthesizers are statistical and (more often) AI models trained to mimic the real data. Once developed, the resulting models are used to create as much synthetic data as you find useful for your use case. 
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: varodrig
body: consider to move this paragraph to another section, since we are not yet explaning what synthesizers or SDV are. **"More on synthesizers below, when looking into the framework (SDV) we selected. For now, synthesizers are statistical and (more often) AI models trained to mimic the real data. Once developed, the resulting models are used to create as much synthetic data as you find useful for your use case."** 

created_at: 2025-02-27T23:45:01Z
author_association: NONE
diff_hunk: @@ -0,0 +1,192 @@
+---
+title: "Synthetic Data Generation with Kubeflow Pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic Data Generation - Why and How?
+
+When creating insights, decisions, and actions from data, the best results come from real data. But accessing real data often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete. 
+
+Synthetic data, uses different techniques outlined below to generate data that mimics the statistical properties of real data, which opens up a good number of use cases. Synthetic data, depending on technique used, not only has similar statistical base properties as the real data, but also preserves correlations between features in the data. 
+
+While the above focuses on speed of development in general, and augmentation of data to improve performance of analytical modes, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is synthetic data, how do we evaluate it?
+
+### Evaluation Criteria for Synthetic Data
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthetic data generation technique (synthesizer) is best for our specific dataset and use case. More on synthesizers below, when looking into the framework (SDV) we selected. For now, synthesizers are statistical and (more often) AI models trained to mimic the real data. Once developed, the resulting models are used to create as much synthetic data as you find useful for your use case. 
+
+We need to ensure a good balance between: 
+
+  - Usability (how useful is the synthetic data for your use case), 
+  - Fidelity (how similar), and 
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: varodrig
body: Since it's a bullet list we can remove "," "and"
created_at: 2025-02-28T00:10:07Z
author_association: NONE
diff_hunk: @@ -0,0 +1,192 @@
+---
+title: "Synthetic Data Generation with Kubeflow Pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic Data Generation - Why and How?
+
+When creating insights, decisions, and actions from data, the best results come from real data. But accessing real data often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete. 
+
+Synthetic data, uses different techniques outlined below to generate data that mimics the statistical properties of real data, which opens up a good number of use cases. Synthetic data, depending on technique used, not only has similar statistical base properties as the real data, but also preserves correlations between features in the data. 
+
+While the above focuses on speed of development in general, and augmentation of data to improve performance of analytical modes, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is synthetic data, how do we evaluate it?
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: varodrig
body: suggestion: to include it inside the next section, to connect it with the related content.
created_at: 2025-02-28T00:11:04Z
author_association: NONE
diff_hunk: @@ -0,0 +1,192 @@
+---
+title: "Synthetic Data Generation with Kubeflow Pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic Data Generation - Why and How?
+
+When creating insights, decisions, and actions from data, the best results come from real data. But accessing real data often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete. 
+
+Synthetic data, uses different techniques outlined below to generate data that mimics the statistical properties of real data, which opens up a good number of use cases. Synthetic data, depending on technique used, not only has similar statistical base properties as the real data, but also preserves correlations between features in the data. 
+
+While the above focuses on speed of development in general, and augmentation of data to improve performance of analytical modes, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is synthetic data, how do we evaluate it?
+
+### Evaluation Criteria for Synthetic Data
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthetic data generation technique (synthesizer) is best for our specific dataset and use case. More on synthesizers below, when looking into the framework (SDV) we selected. For now, synthesizers are statistical and (more often) AI models trained to mimic the real data. Once developed, the resulting models are used to create as much synthetic data as you find useful for your use case. 
+
+We need to ensure a good balance between: 
+
+  - Usability (how useful is the synthetic data for your use case), 
+  - Fidelity (how similar), and 
+  - Privacy (approved level for use case)
+
+For now, we are focusing only on usability and fidelity, using framework-provided measurements for fidelity and workflows described below for usability.
+
+### Frameworks for Creating Synthetic Data 
+
+This post focuses exclusively on open source frameworks.
+Some data cannot be sent to the cloud, so some cloud-based SDG solutions are not always a good fit. 
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: varodrig
body: suggestion: this could be a subsection around "considerations" and then include the data constraints, data privacy issues, etc.  or also could update the title to Considerations when choosing a Synthetic Data Framework
created_at: 2025-02-28T03:12:02Z
author_association: NONE
diff_hunk: @@ -0,0 +1,192 @@
+---
+title: "Synthetic Data Generation with Kubeflow Pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic Data Generation - Why and How?
+
+When creating insights, decisions, and actions from data, the best results come from real data. But accessing real data often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete. 
+
+Synthetic data, uses different techniques outlined below to generate data that mimics the statistical properties of real data, which opens up a good number of use cases. Synthetic data, depending on technique used, not only has similar statistical base properties as the real data, but also preserves correlations between features in the data. 
+
+While the above focuses on speed of development in general, and augmentation of data to improve performance of analytical modes, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is synthetic data, how do we evaluate it?
+
+### Evaluation Criteria for Synthetic Data
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthetic data generation technique (synthesizer) is best for our specific dataset and use case. More on synthesizers below, when looking into the framework (SDV) we selected. For now, synthesizers are statistical and (more often) AI models trained to mimic the real data. Once developed, the resulting models are used to create as much synthetic data as you find useful for your use case. 
+
+We need to ensure a good balance between: 
+
+  - Usability (how useful is the synthetic data for your use case), 
+  - Fidelity (how similar), and 
+  - Privacy (approved level for use case)
+
+For now, we are focusing only on usability and fidelity, using framework-provided measurements for fidelity and workflows described below for usability.
+
+### Frameworks for Creating Synthetic Data 
+
+This post focuses exclusively on open source frameworks.
+Some data cannot be sent to the cloud, so some cloud-based SDG solutions are not always a good fit. 
+For data already in cloud, we can use other cloud-based frameworks.
+Synthesizers are motivated by multiple factors, but in this context, our focus remains on generating synthetic data for on-premise use.
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: varodrig
body: I haven't seen yet the definition about Synthesizers
created_at: 2025-02-28T03:12:28Z
author_association: NONE
diff_hunk: @@ -0,0 +1,192 @@
+---
+title: "Synthetic Data Generation with Kubeflow Pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic Data Generation - Why and How?
+
+When creating insights, decisions, and actions from data, the best results come from real data. But accessing real data often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete. 
+
+Synthetic data, uses different techniques outlined below to generate data that mimics the statistical properties of real data, which opens up a good number of use cases. Synthetic data, depending on technique used, not only has similar statistical base properties as the real data, but also preserves correlations between features in the data. 
+
+While the above focuses on speed of development in general, and augmentation of data to improve performance of analytical modes, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is synthetic data, how do we evaluate it?
+
+### Evaluation Criteria for Synthetic Data
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthetic data generation technique (synthesizer) is best for our specific dataset and use case. More on synthesizers below, when looking into the framework (SDV) we selected. For now, synthesizers are statistical and (more often) AI models trained to mimic the real data. Once developed, the resulting models are used to create as much synthetic data as you find useful for your use case. 
+
+We need to ensure a good balance between: 
+
+  - Usability (how useful is the synthetic data for your use case), 
+  - Fidelity (how similar), and 
+  - Privacy (approved level for use case)
+
+For now, we are focusing only on usability and fidelity, using framework-provided measurements for fidelity and workflows described below for usability.
+
+### Frameworks for Creating Synthetic Data 
+
+This post focuses exclusively on open source frameworks.
+Some data cannot be sent to the cloud, so some cloud-based SDG solutions are not always a good fit. 
+For data already in cloud, we can use other cloud-based frameworks.
+Synthesizers are motivated by multiple factors, but in this context, our focus remains on generating synthetic data for on-premise use.
+
+So, what framework did we (initially) choose? Currently, we are using the open source version of [SDV](https://sdv.dev/), 
+an easy-to-use framework with a strong community and many useful features out-of-the-box (e.g. built-in evaluators, many modeling techniques). 
+The field of synthetic data is evolving rapidly. While we do not aim to cover the latest advancements exhaustively, the use of Foundation models is certainly an area of interest.
+
+**Comments on privacy and privacy preserving techniques**
+
+Ensuring privacy in synthetic data is a non-trivial problem, even if there
+are techniques to ensure levels of privacy, it remains an active area of research. 
+
+*Privacy problems, in synthetic data?*
+
+When creating synthetic data we build models, models from which we generate synthetic data, which can expose privacy problems.
+These models can e.g. be overfitted (close, or fully, memorizing the training data), resulting in leakage of the real data it was trained on. 
+Another challenge lies in handling anomalies. If certain information in the real data stands out and we don't account for it when creating synthetic data, there is a risk of backtracking from the synthetic data to the real.
+A typical example could be 'a very rich person in the dataset' or 'a very rare disease'. 
+The problem here, besides from the privacy part, is that it might be the anomalies we really are looking for.
+
+We are experimenting with various differential privacy strategies, but it is still early days, and we do not focus on them in the examples below.
+
+### The Synthetic Data Vault (SDV)
+
+When you initialize and fit a synthesizer (like GaussianCopulaSynthesizer, CTGANSynthesizer, etc.), it trains a model based on 
+the dataset you provide. This model learns the distribution of the data, capturing the relationships and dependencies between 
+different features in the dataset. Below are the (free) synthesizers provided by SDV that we evaluated on each use case.
+
+The synthesizer doesn't memorize individual records from the dataset. 
+Instead, it tries to learn the underlying statistical patterns, correlations, and distributions present in the data. 
+Each synthesizer does this differently:
+
+- **GaussianCopulaSynthesizer:** Models the distribution of each feature using copulas and marginal distributions. 
+- **CTGANSynthesizer:** Uses a GAN (Generative Adversarial Network) approach, which involves training two neural networks (a generator and a discriminator) to model complex data distributions, especially useful for discrete or mixed-type data.
+- **TVAESynthesizer:** Uses a Variational Autoencoder (VAE) approach to learn a lower-dimensional representation of the data and then generate new samples.
+- **CopulaGANSynthesizer:** Combines elements of copulas and GANs to capture complex dependencies.
+- **PARSynthesizer:** Uses Probabilistic AutoRegressive models to model sequential dependencies in the data.
+
+*There are more synthesizers, also from SDV, but not all are open source.* We used the first four, when evaluating optimal synthesizer for our different use cases.
+
+**Generators - generating new data - on demand**
+
+Once trained, the synthesizer uses the learned model to generate new synthetic data that follows the same statistical properties and 
+distributions as the original dataset, without directly copying any real data points. If you need more data? Just call the generator.
+
+### Our On-Premise Analytics Platform: ARCUS
+
+<div style="display: flex; align-items: center; gap: 20px;">
+  <p>
+    ARCUS is Telia’s advanced on-premise analytics platform, designed to support a wide range of use cases.
+    The platform provides a Kubeflow-based MLOps environment for descriptive, predictive, generative, and (ongoing) agentic AI. 
+    Fully built on open-source, ARCUS integrates a comprehensive stack of components into a unified platform - where Kubernetes is the cornerstone.
+  </p>
+</div>
+
+### Needed environment to create synthetic data
+
+For an efficient, automated selection of the best synthesizer, we need a number of things - from the underlying platform with GPUs and MLOps (Kubeflow).
+
+- Kubeflow pipelines
+- GPU capabilities (for performance and efficiency)
+- Development (IDE) environment (for framework building and running)
+- Modern data platform (MinIO, Airflow) automating the synthetic data generation datasets
+
+#### Parallelism needed
+
+In the (Kube)flows below, we run evaluations in parallel - one for respective synthesizer, followed by a comparison of usability and fidelity scores, selecting the 'winner'.
+
+*Note:* In earlier version of Kubeflow we noticed that the parallelism wasn't acting as expected, waiting for all threads to complete before moving to next step. We had to create a temporary workaround for this, now solved in Kubeflow Pipelines 2.3.0.
+
+Below, we briefly describe the base flow for selecting synthesizer, followed by one use case where we use the resulting data generator for ML development in cloud.
+
+## Exploring the Creation and Usefulness of Synthetic Data
+
+This is what we want to do: we have a use case, the supporting data, and developed ML model.
+
+- How similar is the synthetic data compared to the real data (interesting for e.g. visualization use cases)?
+- How well do the ML models based on synthetic data keep up with ML models based on real data?
+
+**Validation of synthetic data techniques**
+
+1. Create the synthetic data and save the best synthetic data generator. In this step similarity measures are created by the out of the box SDV framework
+2. Create the ML model (in our case classifier model) both on real data and the using the synthetic data. Compare the performance of both models against the same real data testset.
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image-2.png" width="800">
+
+From above, we have an example where the final synthesizer is collected and saved. This step is used in the example  below, exporting the resulting synthetic data generator to cloud.
+
+## Using Synthetic Data Generators to Enable Multiple Environments without Data Transfer
+
+*Below is a usecase where we need to make use of both on-premise and cloud, without moving data to cloud.*
+
+**Problem statement:** 
+
+- Our data cannot be moved from on-premise to cloud.
+- We need extra compute power, in our public cloud environment, to create an ML model for use on-premise.
+- The ML model is to be used on-premise, on new incoming data streams (that cannot be moved to cloud)
+
+**Solution:**
+
+1. Create synthetic data for our on-premise environment use-cases, and - as a side product we save 
+away the synthetic data generator (the pickled model used to create synthetic data). 
+2. Copy the synthetic data generator to cloud
+3. Use the synthetic data generator in the cloud, creating synthetic data for training of an ML model
+4. Copy the ML model on-premise, and use it for new incoming data
+5. Evaluate: Compare the on-premise AI model with the model created in the cloud - against the same test data
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image.png" width="700">
+
+
+**Division of work, what is done on-premise with Kubeflow, and what is done in cloud (AWS SageMaker)?**
+
+##### On-premise
+
+See the above *Validation of synthetic data techniques*.
+
+- Develop the model on real data – for the comparison later with the cloud model.
+- Create synthetic generators, evaluate the generators, and export the best generator to AWS.
+
+##### Cloud
+
+- Use the imported synthetic generator (from on-premise)
+- Create synthetic data using the synthetic data generator
+- Develop the model and determine which synthetic generator is the best
+- Increase the amount of synthetic data, to see if the increase of synthetic data improves model performance (not for sure it will, see below comment)
+- Export model to on-premise
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image-3.png" width="500">
+
+
+In some more detail below.
+
+<img src="/images/2025-02-16-synthetic-data-using-kfp/image-4.png" width="700">
+
+##### On-premise
+
+- Compare real data model against synthetic data model – using real test data.
+
+*In the current examples we see near equivalent performance of the ML models (a few percentage points lower for models created using synthetic data). We experimented with increasing the size of the synthetic dataset, with minor improvements. Augmenting the training data is expected (not tested here) to have more effects when using deep learning algorithms.*
+
+# Summary
+
+Clearly, the above workflows would be very cumbersome to build and maintain without Kubeflow. 
+Our solution is entirely open source, Kubernetes based, and uses Kubeflow and SDV to give us the scalability, robustness, and detailed control that is required.
+
+The area of synthetic data generation is moving fast with the overall AI field. 
+Reports of the increased usage of synthetic data for e.g. LLM training is frequent but the application areas are much greater.
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: tarekabouzeid
body: Yeah sure, an example reference is added.
created_at: 2025-03-01T13:02:22Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,192 @@
+---
+title: "Synthetic Data Generation with Kubeflow Pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic Data Generation - Why and How?
+
+When creating insights, decisions, and actions from data, the best results come from real data. But accessing real data often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete. 
+
+Synthetic data, uses different techniques outlined below to generate data that mimics the statistical properties of real data, which opens up a good number of use cases. Synthetic data, depending on technique used, not only has similar statistical base properties as the real data, but also preserves correlations between features in the data. 
+
+While the above focuses on speed of development in general, and augmentation of data to improve performance of analytical modes, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: akeed
body: Good point. Expanded (and added some more). Updated
created_at: 2025-03-01T15:00:44Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,192 @@
+---
+title: "Synthetic Data Generation with Kubeflow Pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic Data Generation - Why and How?
+
+When creating insights, decisions, and actions from data, the best results come from real data. But accessing real data often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete. 
+
+Synthetic data, uses different techniques outlined below to generate data that mimics the statistical properties of real data, which opens up a good number of use cases. Synthetic data, depending on technique used, not only has similar statistical base properties as the real data, but also preserves correlations between features in the data. 
+
+While the above focuses on speed of development in general, and augmentation of data to improve performance of analytical modes, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is synthetic data, how do we evaluate it?
+
+### Evaluation Criteria for Synthetic Data
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthetic data generation technique (synthesizer) is best for our specific dataset and use case. More on synthesizers below, when looking into the framework (SDV) we selected. For now, synthesizers are statistical and (more often) AI models trained to mimic the real data. Once developed, the resulting models are used to create as much synthetic data as you find useful for your use case. 
+
+We need to ensure a good balance between: 
+
+  - Usability (how useful is the synthetic data for your use case), 
+  - Fidelity (how similar), and 
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: akeed
body: Thanks, updated
created_at: 2025-03-01T15:05:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,192 @@
+---
+title: "Synthetic Data Generation with Kubeflow Pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic Data Generation - Why and How?
+
+When creating insights, decisions, and actions from data, the best results come from real data. But accessing real data often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete. 
+
+Synthetic data, uses different techniques outlined below to generate data that mimics the statistical properties of real data, which opens up a good number of use cases. Synthetic data, depending on technique used, not only has similar statistical base properties as the real data, but also preserves correlations between features in the data. 
+
+While the above focuses on speed of development in general, and augmentation of data to improve performance of analytical modes, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is synthetic data, how do we evaluate it?
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: akeed
body: Thanks, updated
created_at: 2025-03-01T15:09:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,192 @@
+---
+title: "Synthetic Data Generation with Kubeflow Pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic Data Generation - Why and How?
+
+When creating insights, decisions, and actions from data, the best results come from real data. But accessing real data often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete. 
+
+Synthetic data, uses different techniques outlined below to generate data that mimics the statistical properties of real data, which opens up a good number of use cases. Synthetic data, depending on technique used, not only has similar statistical base properties as the real data, but also preserves correlations between features in the data. 
+
+While the above focuses on speed of development in general, and augmentation of data to improve performance of analytical modes, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is synthetic data, how do we evaluate it?
+
+### Evaluation Criteria for Synthetic Data
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthetic data generation technique (synthesizer) is best for our specific dataset and use case. More on synthesizers below, when looking into the framework (SDV) we selected. For now, synthesizers are statistical and (more often) AI models trained to mimic the real data. Once developed, the resulting models are used to create as much synthetic data as you find useful for your use case. 
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: akeed
body: Moved, thanks. Updated
created_at: 2025-03-01T15:21:51Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,192 @@
+---
+title: "Synthetic Data Generation with Kubeflow Pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic Data Generation - Why and How?
+
+When creating insights, decisions, and actions from data, the best results come from real data. But accessing real data often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete. 
+
+Synthetic data, uses different techniques outlined below to generate data that mimics the statistical properties of real data, which opens up a good number of use cases. Synthetic data, depending on technique used, not only has similar statistical base properties as the real data, but also preserves correlations between features in the data. 
+
+While the above focuses on speed of development in general, and augmentation of data to improve performance of analytical modes, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is synthetic data, how do we evaluate it?
+
+### Evaluation Criteria for Synthetic Data
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthetic data generation technique (synthesizer) is best for our specific dataset and use case. More on synthesizers below, when looking into the framework (SDV) we selected. For now, synthesizers are statistical and (more often) AI models trained to mimic the real data. Once developed, the resulting models are used to create as much synthetic data as you find useful for your use case. 
+
+We need to ensure a good balance between: 
+
+  - Usability (how useful is the synthetic data for your use case), 
+  - Fidelity (how similar), and 
+  - Privacy (approved level for use case)
+
+For now, we are focusing only on usability and fidelity, using framework-provided measurements for fidelity and workflows described below for usability.
+
+### Frameworks for Creating Synthetic Data 
+
+This post focuses exclusively on open source frameworks.
+Some data cannot be sent to the cloud, so some cloud-based SDG solutions are not always a good fit. 
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: akeed
body: Thanks, did a change of order of the blow text, better now (I think). Updated
created_at: 2025-03-01T15:55:21Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,192 @@
+---
+title: "Synthetic Data Generation with Kubeflow Pipelines"
+layout: post
+toc: true
+comments: true
+categories: [kfp]
+author: "<a href='https://www.linkedin.com/in/aaked'>Åke Edlund</a>, <a href='https://www.linkedin.com/in/tarekabouzeid91'>Tarek Abouzeid</a>"
+---
+## Synthetic Data Generation - Why and How?
+
+When creating insights, decisions, and actions from data, the best results come from real data. But accessing real data often requires lengthy security and legal processes. The data may also be incomplete, biased, or too small, and during early exploration, we may not even know if it's worth pursuing. While real data is essential for proper evaluation, gaps or limited access frequently hinder progress until the formal process is complete. 
+
+Synthetic data, uses different techniques outlined below to generate data that mimics the statistical properties of real data, which opens up a good number of use cases. Synthetic data, depending on technique used, not only has similar statistical base properties as the real data, but also preserves correlations between features in the data. 
+
+While the above focuses on speed of development in general, and augmentation of data to improve performance of analytical modes, there are more motivations for *creating* (synthetic) data.  Using synthetic data *could* give us new ways to improve on speed of development, handling biases, and more:
+
+- Enhancing privacy and security
+- Increasing data availability
+- Accelerating innovation and experimentation
+- Supporting ethical and responsible AI
+- Improving development speed and time to market
+- Augmenting training datasets for AI models
+
+But, how good is synthetic data, how do we evaluate it?
+
+### Evaluation Criteria for Synthetic Data
+
+There are many aspects to consider when making use of synthetic data, and it is important to evaluate which synthetic data generation technique (synthesizer) is best for our specific dataset and use case. More on synthesizers below, when looking into the framework (SDV) we selected. For now, synthesizers are statistical and (more often) AI models trained to mimic the real data. Once developed, the resulting models are used to create as much synthetic data as you find useful for your use case. 
+
+We need to ensure a good balance between: 
+
+  - Usability (how useful is the synthetic data for your use case), 
+  - Fidelity (how similar), and 
+  - Privacy (approved level for use case)
+
+For now, we are focusing only on usability and fidelity, using framework-provided measurements for fidelity and workflows described below for usability.
+
+### Frameworks for Creating Synthetic Data 
+
+This post focuses exclusively on open source frameworks.
+Some data cannot be sent to the cloud, so some cloud-based SDG solutions are not always a good fit. 
+For data already in cloud, we can use other cloud-based frameworks.
+Synthesizers are motivated by multiple factors, but in this context, our focus remains on generating synthetic data for on-premise use.
path: _posts/2025-02-16-synthetic-data-using-kfp.md
user: login: akeed
body: Good point. Changed order of text to fix that. Updated
created_at: 2025-03-01T15:55:47Z
author_association: CONTRIBUTOR

PR comments:
user: login: tarekabouzeid
created_at: 2025-02-16T17:33:33Z
body: @kubeflow/wg-pipeline-leads  can someone please review ?
cc @akgraner
author_association: MEMBER
user: login: juliusvonkohout
created_at: 2025-02-17T10:58:02Z
body: Maybe post it in the kubeflow pipelines slack channel as well. @hbelmiro @HumairAK @rimolive 
author_association: MEMBER
user: login: tarekabouzeid
created_at: 2025-02-17T12:11:45Z
body: > Maybe post it in the kubeflow pipelines slack channel as well. @hbelmiro @HumairAK @rimolive

Thanks Julius, will do so also
author_association: MEMBER
user: login: tarekabouzeid
created_at: 2025-02-24T18:14:07Z
body: > This is great!!! I left some feedback 🙏

Thank you so much @franciscojavierarceo for your review, i will go through it with @akeed and update the blog. 
author_association: MEMBER
user: login: franciscojavierarceo
created_at: 2025-03-02T11:37:28Z
body: Amazing work here @akeed and @tarekabouzeid! Thanks to everyone for reviewing!

/lgtm /approve
author_association: CONTRIBUTOR
user: login: google-oss-prow[bot]
created_at: 2025-03-02T15:27:13Z
body: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubeflow/blog/pull/159#pullrequestreview-2652753956" title="Approved">franciscojavierarceo</a>*

The full list of commands accepted by this bot can be found [here](https://oss.gprow.dev/command-help?repo=kubeflow%2Fblog).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/kubeflow/blog/blob/master/OWNERS)~~ [franciscojavierarceo]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
author_association: CONTRIBUTOR
user: login: juliusvonkohout
created_at: 2025-03-03T10:11:10Z
body: > Amazing work here @akeed and @tarekabouzeid! Thanks to everyone for reviewing!
> 
> /lgtm /approve

You need to use separate lines for lgtm and approve AFAIK
/lgtm
author_association: MEMBER