Repository: Tracer-Cloud/tracer-client
Pull request #120: Add Otel Fields on Tracer Schema.
Author: login: Iamdavidonuh
Created: 2025-04-15T00:12:28Z | Merged: 2025-04-22T14:38:06Z

## 📌 Summary
<!-- Provide a concise summary of your changes -->

## 🔍 Related Issues/Tickets
<!-- Link to related issues or tickets, e.g., Closes #123 -->

## ✨ Changes Introduced
<!-- Briefly describe the changes in this PR -->

## ✨ Infrastructure Impact
<!-- Briefly describe if there is some impact to our infrastructure -->


## ✅ Checklist
- [ ] Code follows the project's style guidelines
- [ ] I have performed a self-review of my own code
- [ ] I have tested the changes and they work as expected
- [ ] Documentation has been updated if needed
- [ ] Tests have been added or updated

## 🛠️ How to Test
<!-- Provide instructions on how to test your changes -->

## 🚀 Screenshots (if applicable)
<!-- Add screenshots or GIFs to demonstrate the changes -->

## 📌 Additional Notes
<!-- Add any other relevant information -->



Diff:
diff --git a/.github/workflows/build-and-test.yml b/.github/workflows/build-and-test.yml
index 83e6ac91..440eaa1f 100644
--- a/.github/workflows/build-and-test.yml
+++ b/.github/workflows/build-and-test.yml
@@ -72,11 +72,11 @@ jobs:
           AWS_ENDPOINT: "https://s3.us-east-2.amazonaws.com"
           RUST_LOG: "debug"
           RUST_BACKTRACE: 1
-          TRACER_DATABASE_SECRETS_ARN: ${{ secrets.TRACER_DATABASE_SECRETS_ARN }}
+          TRACER_DATABASE_SECRETS_ARN: ${{ secrets.TRACER_DATABASE_LOCAL_SECRETS_ARN }}
           TRACER_DATABASE_NAME: ${{ secrets.TRACER_DATABASE_NAME }}
-          TRACER_DATABASE_HOST: ${{ secrets.TRACER_DATABASE_HOST }}
+          TRACER_DATABASE_HOST: "localhost:5432"
           DATABASE_URL: "postgresql://postgres:postgres@localhost:${{ job.services.postgres.ports[5432] }}/tracer_db" # localhost since VM
-        run: make test-tracer
+        run: ./scripts/migrate.sh "$DATABASE_URL" && make test-tracer
 
   run-integrations:
       name: Build and Test Integrations
@@ -118,12 +118,13 @@ jobs:
 
       - name: Create .env file from GitHub Secrets
         run: |
-          echo "AWS_ACCESS_KEY_ID=${{ secrets.AWS_ACCESS_KEY_ID }}" >> .env
-          echo "AWS_SECRET_ACCESS_KEY=${{ secrets.AWS_SECRET_ACCESS_KEY }}" >> .env
           echo "AWS_REGION=us-east-2" >> .env
           echo "AWS_ENDPOINT=https://s3.us-east-2.amazonaws.com" >> .env
           echo "RUST_LOG=debug" >> .env
           echo "DATABASE_URL=postgres://postgres:postgres@db:5432/tracer_db" >> .env
+          echo "AWS_ACCESS_KEY_ID=${{ secrets.AWS_ACCESS_KEY_ID }}" >> .env
+          echo "AWS_SECRET_ACCESS_KEY=${{ secrets.AWS_SECRET_ACCESS_KEY }}" >> .env
+
 
       - name: Build Test Images
         run: |
@@ -134,3 +135,4 @@ jobs:
 
       - name: Run Parallel tests
         run: make test-parallel
+
diff --git a/.gitignore b/.gitignore
index 3e0de6ce..b8e98fe5 100644
--- a/.gitignore
+++ b/.gitignore
@@ -66,7 +66,7 @@ node_modules/
 
 # Temporary script files
 script-second-script.sh
-/.idea
+**/.idea/
 
 # dev secrets config
-tracer.dev.toml
\ No newline at end of file
+tracer.dev.toml
diff --git a/Cargo.lock b/Cargo.lock
index b07205be..2a7f094b 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -2255,6 +2255,7 @@ name = "integration_tests"
 version = "2025.4.11"
 dependencies = [
  "bollard",
+ "futures-util",
  "sqlx",
  "tokio",
 ]
@@ -4846,7 +4847,9 @@ dependencies = [
  "serde_json",
  "tokio",
  "tracer_aws",
+ "typed-builder",
  "url",
+ "uuid",
 ]
 
 [[package]]
@@ -5009,6 +5012,26 @@ version = "0.2.5"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "e421abadd41a4225275504ea4d6566923418b7f05506fbc9c0fe86ba7396114b"
 
+[[package]]
+name = "typed-builder"
+version = "0.18.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "77739c880e00693faef3d65ea3aad725f196da38b22fdc7ea6ded6e1ce4d3add"
+dependencies = [
+ "typed-builder-macro",
+]
+
+[[package]]
+name = "typed-builder-macro"
+version = "0.18.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "1f718dfaf347dcb5b983bfc87608144b0bad87970aebcbea5ce44d2a30c08e63"
+dependencies = [
+ "proc-macro2",
+ "quote",
+ "syn 2.0.100",
+]
+
 [[package]]
 name = "typenum"
 version = "1.18.0"
diff --git a/Cargo.toml b/Cargo.toml
index 1ad5f707..70f6e81e 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -72,4 +72,6 @@ dotenv = "0.15"
 bollard = "0.18.1"
 assert_cmd = "2.0"
 
+typed-builder = "0.18"
+futures-util = "0.3"
 
diff --git a/docs/DEVELOPMENT.md b/docs/DEVELOPMENT.md
index 932ab815..96c9b3f1 100644
--- a/docs/DEVELOPMENT.md
+++ b/docs/DEVELOPMENT.md
@@ -41,11 +41,7 @@ Then do:
 
 ```bash
 export DATABASE_URL=tracer-cluster-v2-instance-1.cdgizpzxtdp6.us-east-1.rds.amazonaws.com:5432/tracer_db
-<<<<<<< HEAD
 ```
 
 ## Running A Local Environment Test
 init --pipeline-name rnaseq-demo-123 --environment demo --user-operator vincent --pipeline-type rnaseq
-=======
-```
->>>>>>> 5fac3916624f561b6c89993cc687d945bf7b1013
diff --git a/integration_tests/Cargo.toml b/integration_tests/Cargo.toml
index 047024d7..eb27f52d 100644
--- a/integration_tests/Cargo.toml
+++ b/integration_tests/Cargo.toml
@@ -9,6 +9,7 @@ authors = { workspace = true }
 [dependencies]
 sqlx = { workspace = true }
 tokio = { workspace = true }
+futures-util = {workspace = true }
 
 [dev-dependencies]
 bollard = { workspace = true }
diff --git a/integration_tests/tests/common/mod.rs b/integration_tests/tests/common/mod.rs
index 1301efcf..7f1fec24 100644
--- a/integration_tests/tests/common/mod.rs
+++ b/integration_tests/tests/common/mod.rs
@@ -1,7 +1,12 @@
 use bollard::container::{InspectContainerOptions, ListContainersOptions};
+use bollard::container::{LogOutput, LogsOptions};
 use bollard::Docker;
+use futures_util::stream::StreamExt;
 use sqlx::PgPool;
+use std::io::Write;
 use std::process::Command;
+use tokio::io::{AsyncBufReadExt, BufReader};
+use tokio::process::Command as TokioCommand;
 use tokio::time::{sleep, Duration};
 
 pub async fn monitor_container(docker: &Docker, container_prefix: &str) {
@@ -137,3 +142,150 @@ pub async fn setup_db(db_url: &str) -> PgPool {
 
     pool
 }
+
+pub async fn print_all_container_logs(docker: &Docker) {
+    let containers = docker
+        .list_containers(Some(ListContainersOptions::<String> {
+            all: true,
+            ..Default::default()
+        }))
+        .await
+        .expect("Failed to list containers");
+
+    for container in containers {
+        if let Some(names) = container.names {
+            let name = names
+                .first()
+                .map(|n| n.trim_start_matches('/'))
+                .unwrap_or("unknown");
+
+            let options = Some(LogsOptions::<String> {
+                stdout: true,
+                stderr: true,
+                follow: false,
+                tail: "all".into(),
+                ..Default::default()
+            });
+
+            println!("\n📦 Logs for container: `{}`", name);
+
+            let mut logs = docker.logs(name, options);
+
+            while let Some(log_result) = logs.next().await {
+                match log_result {
+                    Ok(LogOutput::StdOut { message }) | Ok(LogOutput::StdErr { message }) => {
+                        print!("{}", String::from_utf8_lossy(&message));
+                        std::io::stdout().flush().unwrap();
+                    }
+                    Ok(_) => {}
+                    Err(err) => {
+                        eprintln!("❌ Error reading logs for `{}`: {}", name, err);
+                        break;
+                    }
+                }
+            }
+        }
+    }
+}
+
+pub fn dump_container_file(container: &str, file_path: &str) {
+    let output = Command::new("docker")
+        .arg("exec")
+        .arg(container)
+        .arg("cat")
+        .arg(file_path)
+        .output()
+        .expect("Failed to exec into container");
+
+    if output.status.success() {
+        println!(
+            "📄 Contents of `{}` in container `{}`:\n{}",
+            file_path,
+            container,
+            String::from_utf8_lossy(&output.stdout)
+        );
+    } else {
+        eprintln!(
+            "❌ Error reading `{}`: {}",
+            file_path,
+            String::from_utf8_lossy(&output.stderr)
+        );
+    }
+}
+
+pub async fn dump_container_file_for_all_matching(
+    docker: &Docker,
+    container_prefix: &str,
+    file_path: &str,
+) {
+    let containers = docker
+        .list_containers(Some(ListContainersOptions::<String> {
+            all: true,
+            ..Default::default()
+        }))
+        .await
+        .expect("Failed to list containers");
+
+    let mut tasks = vec![];
+
+    for container in containers {
+        let container_name = container
+            .names
+            .as_ref()
+            .and_then(|names| names.first())
+            .map(|name| name.trim_start_matches('/').to_string())
+            .unwrap_or_else(|| "unknown".to_string());
+
+        let image_name = container.image.unwrap_or_default();
+
+        if container_name.contains(container_prefix) || image_name.contains(container_prefix) {
+            let name = container_name.clone();
+            let path = file_path.to_string();
+            let docker = docker.clone();
+
+            tasks.push(tokio::spawn(async move {
+                loop {
+                    let is_running = match docker
+                        .inspect_container(&name, None)
+                        .await
+                        .ok()
+                        .and_then(|info| info.state)
+                    {
+                        Some(state) => state.running.unwrap_or(false),
+                        None => false,
+                    };
+
+                    if !is_running {
+                        break;
+                    }
+
+                    let mut cmd = TokioCommand::new("docker")
+                        .arg("exec")
+                        .arg(&name)
+                        .arg("cat")
+                        .arg(&path)
+                        .stdout(std::process::Stdio::piped())
+                        .spawn()
+                        .expect("Failed to exec docker");
+
+                    if let Some(stdout) = cmd.stdout.take() {
+                        let mut reader = BufReader::new(stdout).lines();
+
+                        while let Ok(Some(line)) = reader.next_line().await {
+                            println!("📄 [{}:{}] {}", name, path, line);
+                        }
+                    }
+
+                    sleep(Duration::from_secs(2)).await;
+                }
+
+                println!("✅ Done streaming `{}` for container `{}`", path, name);
+            }));
+        }
+    }
+
+    // Wait for all tasks to complete
+    for task in tasks {
+        let _ = task.await;
+    }
+}
diff --git a/integration_tests/tests/parallel.rs b/integration_tests/tests/parallel.rs
index efb8537f..82af8492 100644
--- a/integration_tests/tests/parallel.rs
+++ b/integration_tests/tests/parallel.rs
@@ -30,7 +30,7 @@ async fn test_parallel_mode_works() {
 async fn query_and_assert_parallel_mode(pool: &PgPool, run_name: &str) {
     let tools_tracked: Vec<(i64,)> = sqlx::query_as(
         r#"
-            SELECT COUNT(DISTINCT data->'attributes'->'system_properties'->>'hostname') AS unique_hosts
+            SELECT COUNT(DISTINCT resource_attributes->>'hostname') AS unique_hosts
             FROM batch_jobs_logs
             WHERE run_name = $1;
         "#,
diff --git a/integration_tests/tests/simple_queries.rs b/integration_tests/tests/simple_queries.rs
index 222a626a..16223274 100644
--- a/integration_tests/tests/simple_queries.rs
+++ b/integration_tests/tests/simple_queries.rs
@@ -17,27 +17,45 @@ async fn test_queries_works() {
     // Step 2: Monitor the container and wait for it to finish
     let docker = Docker::connect_with_local_defaults().expect("Failed to connect to Docker");
 
+    let log_handle = tokio::spawn({
+        let docker = docker.clone();
+        async move {
+            common::print_all_container_logs(&docker).await;
+            common::dump_container_file_for_all_matching(
+                &docker,
+                container_name,
+                "/tmp/tracer/tracerd.out",
+            )
+            .await;
+            common::dump_container_file_for_all_matching(
+                &docker,
+                container_name,
+                "/tmp/tracer/tracerd.err",
+            )
+            .await;
+        }
+    });
+
     common::monitor_container(&docker, container_name).await;
 
     // Step 3: Query the database and make assertions
     let run_name = "test-tag";
+    let _ = log_handle.await;
 
     query_and_assert_tool_tracked(&pool, run_name).await;
 
     query_datasets_processed(&pool, run_name).await;
-
-    common::end_docker_compose(container_name).await;
 }
 
 async fn query_and_assert_tool_tracked(pool: &PgPool, run_name: &str) {
     let tools_tracked: Vec<(String,)> = sqlx::query_as(
         r#"
-            SELECT DISTINCT(data->'attributes'->'process'->>'tool_name') AS tool_name
+            SELECT DISTINCT(attributes->>'tool_name') AS tool_name
             FROM batch_jobs_logs
             WHERE 
             run_name = $1
             AND
-            data->'attributes'->'process'->>'tool_name' IS NOT NULL;
+            attributes ->> 'tool_name' IS NOT NULL;
         "#,
     )
     .bind(run_name)
@@ -52,11 +70,11 @@ async fn query_and_assert_tool_tracked(pool: &PgPool, run_name: &str) {
 }
 
 async fn query_datasets_processed(pool: &PgPool, run_name: &str) {
-    let tools_tracked: Vec<(String, i64)> = sqlx::query_as(
+    let datasets_tracked: Vec<(String, i64)> = sqlx::query_as(
         r#"
             SELECT 
                 process_status,
-                MAX((data->'attributes'->'process_dataset_stats'->>'total')::BIGINT) AS total_samples
+                MAX((attributes ->> 'total')::BIGINT) AS total_samples
             FROM batch_jobs_logs
             WHERE process_status = 'datasets_in_process'
             AND run_name = $1
@@ -67,9 +85,9 @@ async fn query_datasets_processed(pool: &PgPool, run_name: &str) {
     .fetch_all(pool)
     .await
     .expect("failed ");
-    assert_eq!(tools_tracked.len(), 1);
+    assert_eq!(datasets_tracked.len(), 1);
 
-    let total_samples = tools_tracked.first().unwrap().1;
+    let total_samples = datasets_tracked.first().unwrap().1;
 
     assert_eq!(total_samples, 3)
 }
diff --git a/migrations/20250414175511_new_schema_update.down.sql b/migrations/20250414175511_new_schema_update.down.sql
new file mode 100644
index 00000000..9f33c6d2
--- /dev/null
+++ b/migrations/20250414175511_new_schema_update.down.sql
@@ -0,0 +1,31 @@
+-- Drop otel_logs table
+DROP TABLE IF EXISTS batch_jobs_logs;
+
+-- Recreate batch_jobs_logs as it existed before
+CREATE TABLE IF NOT EXISTS batch_jobs_logs (
+    id SERIAL PRIMARY KEY,
+    data JSONB NOT NULL,
+    job_id TEXT,
+    creation_date TIMESTAMP DEFAULT NOW(),
+    run_name TEXT,
+    run_id TEXT,
+    pipeline_name TEXT,
+    nextflow_session_uuid TEXT,
+    job_ids TEXT[],
+    tags JSONB,
+    event_timestamp TIMESTAMP,
+    ec2_cost_per_hour FLOAT,
+    cpu_usage FLOAT,
+    mem_used FLOAT,
+    processed_dataset INT,
+    process_status TEXT
+);
+
+-- Rebuild indexes if needed (can be omitted if you don’t need rollback performance)
+CREATE INDEX IF NOT EXISTS idx_batch_jobs_logs_metrics
+    ON batch_jobs_logs (job_id, pipeline_name, tags, event_timestamp, ec2_cost_per_hour, cpu_usage, mem_used, processed_dataset);
+
+CREATE INDEX IF NOT EXISTS idx_batch_jobs_logs_process_status
+    ON batch_jobs_logs (process_status);
+
+ANALYZE batch_jobs_logs;
\ No newline at end of file
diff --git a/migrations/20250414175511_new_schema_update.up.sql b/migrations/20250414175511_new_schema_update.up.sql
new file mode 100644
index 00000000..eb30abf6
--- /dev/null
+++ b/migrations/20250414175511_new_schema_update.up.sql
@@ -0,0 +1,64 @@
+-- Drop the old table
+DROP TABLE IF EXISTS batch_jobs_logs;
+
+-- Create the new batch_jobs_logs table
+CREATE TABLE IF NOT EXISTS batch_jobs_logs (
+    event_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+
+    -- Core OTel fields
+    timestamp TIMESTAMPTZ NOT NULL,
+    body TEXT NOT NULL,
+    severity_text TEXT,
+    severity_number SMALLINT,
+    trace_id TEXT,
+    span_id TEXT,
+
+    -- Instrumentation metadata
+    source_type TEXT,
+    instrumentation_version TEXT,
+    instrumentation_type TEXT,
+
+    -- Tags (flattened from JSON)
+    environment TEXT,
+    pipeline_type TEXT,
+    user_operator TEXT,
+    department TEXT,
+    organization_id TEXT,
+
+    -- Pipeline context 
+    run_id TEXT,
+    run_name TEXT,
+    pipeline_name TEXT,
+
+    -- Execution tracing (generalized)
+    job_id TEXT,
+    parent_job_id TEXT,
+    child_job_ids TEXT[],
+    workflow_engine TEXT,
+
+    -- Performance/metrics fields
+    ec2_cost_per_hour FLOAT,
+    cpu_usage FLOAT,
+    mem_used FLOAT,
+    processed_dataset INT,
+    process_status TEXT,
+
+    -- Full structured logs and metadata
+    attributes JSONB,
+    resource_attributes JSONB,
+    tags JSONB
+);
+
+-- Indexes
+CREATE INDEX IF NOT EXISTS idx_batch_jobs_logs_timestamp ON batch_jobs_logs (timestamp);
+CREATE INDEX IF NOT EXISTS idx_batch_jobs_logs_run_pipeline ON batch_jobs_logs (run_id, pipeline_name);
+CREATE INDEX IF NOT EXISTS idx_batch_jobs_logs_job_trace ON batch_jobs_logs (job_id, parent_job_id, workflow_engine);
+CREATE INDEX IF NOT EXISTS idx_batch_jobs_logs_metrics ON batch_jobs_logs (
+    cpu_usage,
+    mem_used,
+    ec2_cost_per_hour,
+    processed_dataset
+);
+CREATE INDEX IF NOT EXISTS idx_batch_jobs_logs_process_status ON batch_jobs_logs (process_status);
+
+ANALYZE batch_jobs_logs;
\ No newline at end of file
diff --git a/src/client/src/exporters/db.rs b/src/client/src/exporters/db.rs
index 21cc814d..31565b99 100644
--- a/src/client/src/exporters/db.rs
+++ b/src/client/src/exporters/db.rs
@@ -1,22 +1,17 @@
 use anyhow::{Context, Result};
-use chrono::{DateTime, Utc};
 use log::info;
 use sqlx::pool::PoolOptions;
-use sqlx::types::Json;
 use sqlx::{PgPool, Postgres, QueryBuilder};
-use std::time::Instant;
+use tracer_common::event::{Event, EventInsert};
 
 use crate::config_manager::Config;
-
 use percent_encoding::{utf8_percent_encode, NON_ALPHANUMERIC};
-use serde_json::Value;
+
 use sqlx::query_builder::Separated;
 use tracer_aws::config::SecretsClient;
 use tracer_aws::types::secrets::DatabaseAuth;
-use tracer_common::event::attributes::EventAttributes;
-use tracer_common::event::Event;
-use tracing::debug;
 
+use tracing::debug;
 const BIND_LIMIT: usize = 65535;
 
 pub struct AuroraClient {
@@ -29,90 +24,6 @@ impl AuroraClient {
     }
 }
 
-struct EventInsert {
-    json_data: Json<Value>,
-    job_id: Option<String>,
-    run_name: String,
-    run_id: String,
-    pipeline_name: String,
-    nextflow_session_uuid: Option<String>,
-    job_ids: Vec<String>,
-    tags_json: Json<Value>,
-    event_timestamp: DateTime<Utc>,
-    ec2_cost_per_hour: Option<f64>,
-    cpu_usage: Option<f64>,
-    mem_used: Option<f64>,
-    processed_dataset: Option<i32>,
-    process_status: String,
-}
-
-impl EventInsert {
-    pub fn try_new(
-        event: &Event,
-        run_name: String,
-        run_id: String,
-        pipeline_name: String,
-        process_status: String,
-    ) -> Result<Self> {
-        let json_data = Json(serde_json::to_value(event)?);
-        let (nextflow_session_uuid, job_ids, event_timestamp) = match &event.attributes {
-            Some(EventAttributes::NextflowLog(log)) => (
-                log.session_uuid.clone(),                 // Option<String>
-                log.jobs_ids.clone().unwrap_or_default(), // Vec<String>
-                event.timestamp,                          // Use event timestamp directly
-            ),
-            _ => (None, Vec::new(), event.timestamp),
-        };
-
-        // Extract system metrics for CPU and memory usage
-        let (cpu_usage, mem_used, job_id) = match &event.attributes {
-            Some(EventAttributes::SystemMetric(metric)) => (
-                Some(metric.system_cpu_utilization as f64),
-                Some(metric.system_memory_used as f64),
-                None,
-            ),
-            Some(EventAttributes::Process(process)) => (
-                Some(process.process_cpu_utilization as f64),
-                Some(process.process_memory_usage as f64),
-                Some(process.job_id.clone().unwrap_or_default()),
-            ),
-            _ => (None, None, None),
-        };
-
-        // Extract EC2 cost per hour from system properties
-        let ec2_cost_per_hour = match &event.attributes {
-            Some(EventAttributes::SystemProperties(props)) => props.ec2_cost_per_hour,
-            _ => None,
-        };
-
-        // Extract processed dataset count
-        let processed_dataset = match &event.attributes {
-            Some(EventAttributes::ProcessDatasetStats(stats)) => Some(stats.total as i32),
-            _ => None,
-        };
-
-        // Convert tags to JSON value
-        let tags_json = Json(serde_json::to_value(&event.tags)?);
-
-        Ok(Self {
-            json_data,
-            job_id,
-            run_name,
-            run_id,
-            pipeline_name,
-            nextflow_session_uuid,
-            job_ids,
-            cpu_usage,
-            mem_used,
-            ec2_cost_per_hour,
-            processed_dataset,
-            tags_json,
-            event_timestamp,
-            process_status,
-        })
-    }
-}
-
 impl AuroraClient {
     pub async fn try_new(config: &Config, pool_size: Option<u32>) -> Result<Self> {
         let secrets_client = SecretsClient::new(config.aws_init_type.clone()).await;
@@ -165,30 +76,52 @@ impl AuroraClient {
         pipeline_name: &str,
         data: impl IntoIterator<Item = &Event>,
     ) -> Result<()> {
-        let now = Instant::now();
+        let now = std::time::Instant::now();
 
         const QUERY: &str = "INSERT INTO batch_jobs_logs (
-            data, job_id, run_name, run_id, pipeline_name, nextflow_session_uuid, job_ids,
-            tags, event_timestamp, ec2_cost_per_hour, cpu_usage, mem_used, processed_dataset,
-            process_status
-            )"; // when updating query, also update params
-        const PARAMS: usize = 13;
+                timestamp, body, severity_text, severity_number,
+                trace_id, span_id,
+                source_type, instrumentation_version, instrumentation_type,
+                environment, pipeline_type, user_operator, organization_id, department,
+                run_id, run_name, pipeline_name,
+                job_id, parent_job_id, child_job_ids, workflow_engine,
+                ec2_cost_per_hour, cpu_usage, mem_used, processed_dataset,
+                process_status,
+                attributes, resource_attributes, tags
+            )";
+        // when updating query, also update params
+        const PARAMS: usize = 29;
 
         fn _push_tuple(mut b: Separated<Postgres, &str>, event: EventInsert) {
-            b.push_bind(event.json_data)
-                .push_bind(event.job_id)
-                .push_bind(event.run_name)
+            b.push_bind(event.event_timestamp.naive_utc())
+                .push_bind(event.body)
+                .push_bind(event.severity_text)
+                .push_bind(event.severity_number)
+                .push_bind(event.trace_id.or_else(|| Some(event.run_id.clone())))
+                .push_bind(event.span_id)
+                .push_bind(event.source_type)
+                .push_bind(event.instrumentation_version)
+                .push_bind(event.instrumentation_type)
+                .push_bind(event.environment)
+                .push_bind(event.pipeline_type)
+                .push_bind(event.user_operator)
+                .push_bind(event.organization_id)
+                .push_bind(event.department)
                 .push_bind(event.run_id)
+                .push_bind(event.run_name)
                 .push_bind(event.pipeline_name)
-                .push_bind(event.nextflow_session_uuid)
-                .push_bind(event.job_ids)
-                .push_bind(event.tags_json)
-                .push_bind(event.event_timestamp.naive_utc())
+                .push_bind(event.job_id)
+                .push_bind(event.parent_job_id)
+                .push_bind(event.child_job_ids)
+                .push_bind(event.workflow_engine)
                 .push_bind(event.ec2_cost_per_hour)
                 .push_bind(event.cpu_usage)
                 .push_bind(event.mem_used)
                 .push_bind(event.processed_dataset)
-                .push_bind(event.process_status);
+                .push_bind(event.process_status)
+                .push_bind(event.attributes)
+                .push_bind(event.resource_attributes)
+                .push_bind(event.tags);
         }
         // there's an alternative, much more efficient way to push values
         // https://github.com/launchbadge/sqlx/blob/main/FAQ.md#how-can-i-bind-an-array-to-a-values-clause-how-can-i-do-bulk-inserts
@@ -196,55 +129,38 @@ impl AuroraClient {
         // see https://github.com/launchbadge/sqlx/issues/1945
 
         info!(
-            "Inserting row for run_name: {}, pipeline_name: {}",
-            run_name, pipeline_name
+            "Inserting row for run_name: {}, pipeline_name: {}, run_id: {}",
+            run_name, pipeline_name, run_id
         );
 
         let mut builder = QueryBuilder::new(QUERY);
 
         let mut data: Vec<_> = data
             .into_iter()
-            .map(|e| {
-                EventInsert::try_new(
-                    e,
-                    run_name.to_string(),
-                    run_id.to_string(),
-                    pipeline_name.to_string(),
-                    e.process_status.as_str().to_string(),
-                )
-            })
-            .collect::<Result<Vec<_>>>()?;
+            .cloned()
+            .filter_map(|e| e.try_into().ok())
+            .collect();
 
         let rows_affected = match data.len() {
             0 => {
                 debug!("No data to insert");
                 return Ok(());
             }
-
             x if x * PARAMS >= BIND_LIMIT => {
-                debug!("Using transaction to insert data");
+                debug!("Chunked insert with transaction due to bind limit");
                 let mut transaction = self.pool.begin().await?;
-
                 let mut rows_affected = 0;
 
                 while !data.is_empty() {
-                    let chunk = data.split_off(data.len().min(BIND_LIMIT / PARAMS));
+                    let chunk: Vec<_> = data.split_off(data.len().min(BIND_LIMIT / PARAMS));
                     let query = builder.push_values(chunk, _push_tuple).build();
-
-                    let chunk_rows_affected = query
-                        .execute(&mut *transaction)
-                        .await
-                        .context("Failed to insert event into database")?
-                        .rows_affected();
+                    rows_affected += query.execute(&mut *transaction).await?.rows_affected();
                     builder.reset();
-
-                    rows_affected += chunk_rows_affected;
                 }
 
                 transaction.commit().await?;
                 rows_affected
             }
-
             _ => {
                 debug!("Inserting data without transaction");
                 builder.push_values(data, _push_tuple);
diff --git a/src/client/src/tracer_client.rs b/src/client/src/tracer_client.rs
index b200a4b5..ec2fe359 100644
--- a/src/client/src/tracer_client.rs
+++ b/src/client/src/tracer_client.rs
@@ -385,13 +385,6 @@ impl TracerClient {
             .await
     }
 
-    pub async fn poll_nextflow_log(&mut self) -> Result<()> {
-        self.process_watcher
-            .get_nextflow_log_watcher_mut()
-            .poll_nextflow_log(&mut self.logs)
-            .await
-    }
-
     pub fn refresh_sysinfo(&mut self) {
         self.system.refresh_all();
     }
@@ -501,7 +494,7 @@ mod tests {
         assert!(res.is_ok());
 
         // Prepare the SQL query
-        let query = "SELECT data, run_name FROM batch_jobs_logs WHERE run_name = $1";
+        let query = "SELECT attributes, run_name FROM batch_jobs_logs WHERE run_name = $1";
 
         let db_client = client.db_client.get_pool();
 
diff --git a/src/common/Cargo.toml b/src/common/Cargo.toml
index e39d2229..17347557 100644
--- a/src/common/Cargo.toml
+++ b/src/common/Cargo.toml
@@ -17,4 +17,7 @@ serde = { workspace = true }
 serde_json = { workspace = true }
 tokio = { workspace = true }
 url = { workspace = true }
-tracer_aws = { workspace = true }
\ No newline at end of file
+tracer_aws = { workspace = true }
+uuid = { workspace = true }
+
+typed-builder = { workspace = true}
diff --git a/src/common/src/event/attributes/mod.rs b/src/common/src/event/attributes/mod.rs
index f22ace65..0e0dc800 100644
--- a/src/common/src/event/attributes/mod.rs
+++ b/src/common/src/event/attributes/mod.rs
@@ -1,4 +1,3 @@
-use crate::event::attributes::system_metrics::NextflowLog;
 use process::{CompletedProcess, DataSetsProcessed, ProcessProperties};
 use syslog::SyslogProperties;
 use system_metrics::{SystemMetric, SystemProperties};
@@ -16,7 +15,4 @@ pub enum EventAttributes {
     Syslog(SyslogProperties),
     SystemProperties(SystemProperties),
     ProcessDatasetStats(DataSetsProcessed),
-    NextflowLog(NextflowLog),
-    // TODO: take out when done with demo
-    Other(serde_json::Value),
 }
diff --git a/src/common/src/event/attributes/process.rs b/src/common/src/event/attributes/process.rs
index 33a37349..c819d4a7 100644
--- a/src/common/src/event/attributes/process.rs
+++ b/src/common/src/event/attributes/process.rs
@@ -30,6 +30,7 @@ pub struct ProcessProperties {
     pub container_id: Option<String>,
     pub job_id: Option<String>,
     pub working_directory: Option<String>,
+    pub trace_id: Option<String>,
 }
 
 #[derive(Debug, Clone, Serialize, Deserialize)]
@@ -43,4 +44,5 @@ pub struct CompletedProcess {
 pub struct DataSetsProcessed {
     pub datasets: String,
     pub total: u64,
+    pub trace_id: Option<String>,
 }
diff --git a/src/common/src/event/attributes/system_metrics.rs b/src/common/src/event/attributes/system_metrics.rs
index 571fd853..7f905260 100644
--- a/src/common/src/event/attributes/system_metrics.rs
+++ b/src/common/src/event/attributes/system_metrics.rs
@@ -39,9 +39,3 @@ pub struct SystemProperties {
     // cost analysis
     pub ec2_cost_per_hour: Option<f64>,
 }
-
-#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]
-pub struct NextflowLog {
-    pub session_uuid: Option<String>,
-    pub jobs_ids: Option<Vec<String>>,
-}
diff --git a/src/common/src/event/mod.rs b/src/common/src/event/mod.rs
index 28e2e40e..94ff7c3a 100644
--- a/src/common/src/event/mod.rs
+++ b/src/common/src/event/mod.rs
@@ -5,6 +5,13 @@ use crate::pipeline_tags::PipelineTags;
 use chrono::serde::ts_seconds;
 use chrono::{DateTime, Utc};
 use serde::{Deserialize, Serialize};
+use serde_json::{json, Value};
+use typed_builder::TypedBuilder;
+use uuid::Uuid;
+
+use anyhow::Context;
+
+use std::convert::TryFrom;
 
 #[derive(serde::Serialize, serde::Deserialize, Debug, Clone, PartialEq)]
 #[serde(rename_all = "snake_case")]
@@ -33,35 +40,46 @@ pub enum ProcessStatus {
     #[serde(rename = "datasets_in_process")]
     DataSamplesEvent,
     TestEvent, // Added TestEvent variant
-    NextflowLogEvent,
 }
 
-impl ProcessStatus {
-    pub fn as_str(&self) -> &'static str {
+impl std::fmt::Display for ProcessStatus {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
         match self {
-            ProcessStatus::NewRun => "new_run",
-            ProcessStatus::FinishedRun => "finished_run",
-            ProcessStatus::ToolExecution => "tool_execution",
-            ProcessStatus::FinishedToolExecution => "finished_tool_execution",
-            ProcessStatus::MetricEvent => "metric_event",
-            ProcessStatus::SyslogEvent => "syslog_event",
-            ProcessStatus::ToolMetricEvent => "tool_metric_event",
-            ProcessStatus::TestEvent => "test_event", // Handle TestEvent
-            ProcessStatus::RunStatusMessage => "run_status_message",
-            ProcessStatus::Alert => "alert",
-            ProcessStatus::DataSamplesEvent => "datasets_in_process",
-            ProcessStatus::NextflowLogEvent => "nextflow_log_event",
+            ProcessStatus::NewRun => write!(f, "new_run"),
+            ProcessStatus::FinishedRun => write!(f, "finished_run"),
+            ProcessStatus::ToolExecution => write!(f, "tool_execution"),
+            ProcessStatus::FinishedToolExecution => write!(f, "finished_tool_execution"),
+            ProcessStatus::ToolMetricEvent => write!(f, "tool_metric_event"),
+            ProcessStatus::MetricEvent => write!(f, "metric_event"),
+            ProcessStatus::SyslogEvent => write!(f, "syslog_event"),
+            ProcessStatus::RunStatusMessage => write!(f, "run_status_message"),
+            ProcessStatus::Alert => write!(f, "alert"),
+            ProcessStatus::DataSamplesEvent => write!(f, "datasets_in_process"),
+            ProcessStatus::TestEvent => write!(f, "test_event"),
         }
     }
 }
-#[derive(serde::Serialize, serde::Deserialize, Debug, Clone)]
+
+fn default_span_id() -> Option<String> {
+    Some(Uuid::new_v4().to_string())
+}
+
+#[derive(Serialize, Deserialize, Debug, Clone, TypedBuilder)]
+#[builder(field_defaults(default))]
 pub struct Event {
     #[serde(with = "ts_seconds")]
     pub timestamp: DateTime<Utc>,
-    pub message: String,
 
+    #[builder(setter(into))]
+    pub body: String,
+
+    #[builder(default = EventType::ProcessStatus)]
     pub event_type: EventType,
+
+    #[builder(default = ProcessType::Pipeline)]
     pub process_type: ProcessType,
+
+    #[builder(default = ProcessStatus::TestEvent)]
     pub process_status: ProcessStatus,
 
     pub pipeline_name: Option<String>,
@@ -69,4 +87,139 @@ pub struct Event {
     pub run_id: Option<String>,
     pub attributes: Option<EventAttributes>,
     pub tags: Option<PipelineTags>,
+
+    pub severity_text: Option<String>,
+    pub severity_number: Option<u8>,
+    pub trace_id: Option<String>,
+
+    #[builder(default = default_span_id())]
+    pub span_id: Option<String>,
+}
+
+// TODO: would be removed in next pr
+pub struct EventInsert {
+    pub event_timestamp: DateTime<Utc>,
+    pub body: String,
+    pub severity_text: Option<String>,
+    pub severity_number: Option<i16>,
+    pub trace_id: Option<String>,
+    pub span_id: Option<String>,
+
+    pub source_type: String,
+    pub instrumentation_version: Option<String>,
+    pub instrumentation_type: Option<String>,
+    pub environment: Option<String>,
+    pub pipeline_type: Option<String>,
+    pub user_operator: Option<String>,
+    pub organization_id: Option<String>,
+    pub department: Option<String>,
+
+    pub run_id: String,
+    pub run_name: String,
+    pub pipeline_name: String,
+    pub job_id: Option<String>,
+    pub parent_job_id: Option<String>,
+    pub child_job_ids: Option<Vec<String>>,
+    pub workflow_engine: Option<String>,
+
+    pub ec2_cost_per_hour: Option<f64>,
+    pub cpu_usage: Option<f32>,
+    pub mem_used: Option<f64>,
+    pub processed_dataset: Option<i32>,
+    pub process_status: String,
+
+    pub attributes: Value,
+    pub resource_attributes: Value,
+    pub tags: Value,
+}
+
+impl TryFrom<Event> for EventInsert {
+    type Error = anyhow::Error;
+
+    fn try_from(event: Event) -> anyhow::Result<Self> {
+        let mut attributes = json!({});
+        let mut resource_attributes = json!({});
+        let mut job_id = None;
+        let parent_job_id = None;
+        let child_job_ids = None;
+        let workflow_engine = None;
+        let mut cpu_usage = None;
+        let mut mem_used = None;
+        let mut ec2_cost_per_hour = None;
+        let mut processed_dataset = None;
+        let mut trace_id = None;
+
+        if let Some(attr) = &event.attributes {
+            match attr {
+                EventAttributes::Process(p) => {
+                    cpu_usage = Some(p.process_cpu_utilization);
+                    mem_used = Some(p.process_memory_usage as f64);
+                    job_id = p.job_id.clone();
+                    trace_id = p.trace_id.clone();
+                    attributes = serde_json::to_value(p)
+                        .context("Failed to serialize Process attributes")?;
+                }
+                EventAttributes::SystemMetric(m) => {
+                    cpu_usage = Some(m.system_cpu_utilization);
+                    mem_used = Some(m.system_memory_used as f64);
+                    attributes = serde_json::to_value(m)
+                        .context("Failed to serialize SystemMetric attributes")?;
+                }
+                EventAttributes::SystemProperties(p) => {
+                    ec2_cost_per_hour = p.ec2_cost_per_hour;
+                    resource_attributes =
+                        serde_json::to_value(p).context("Failed to serialize SystemProperties")?;
+                }
+                EventAttributes::ProcessDatasetStats(d) => {
+                    processed_dataset = Some(d.total as i32);
+                    attributes = serde_json::to_value(d)
+                        .context("Failed to serialize ProcessDatasetStats")?;
+                    trace_id = d.trace_id.clone()
+                }
+                EventAttributes::Syslog(s) => {
+                    attributes =
+                        serde_json::to_value(s).context("Failed to serialize Syslog attributes")?;
+                }
+                _ => {}
+            }
+        }
+
+        let tags = event.tags.clone();
+
+        Ok(EventInsert {
+            event_timestamp: event.timestamp,
+            body: event.body,
+            severity_text: event.severity_text,
+            severity_number: event.severity_number.map(|v| v as i16),
+            trace_id: trace_id.or_else(|| event.run_id.clone()),
+            span_id: event.span_id,
+
+            source_type: "tracer-daemon".into(),
+            instrumentation_version: option_env!("CARGO_PKG_VERSION").map(|s| s.to_string()),
+            instrumentation_type: Some("TRACER_DAEMON".into()),
+            environment: tags.clone().map(|t| t.environment),
+            pipeline_type: tags.clone().map(|t| t.pipeline_type),
+            user_operator: tags.clone().map(|t| t.user_operator),
+            organization_id: tags.clone().map(|t| t.organization_id).unwrap_or_default(),
+            department: tags.clone().map(|t| t.department),
+
+            run_id: event.run_id.unwrap_or_default(),
+            run_name: event.run_name.unwrap_or_default(),
+            pipeline_name: event.pipeline_name.unwrap_or_default(),
+            job_id,
+            parent_job_id,
+            child_job_ids,
+            workflow_engine,
+
+            ec2_cost_per_hour,
+            cpu_usage,
+            mem_used,
+            processed_dataset,
+            process_status: event.process_status.to_string(),
+
+            attributes,
+            resource_attributes,
+            tags: serde_json::to_value(tags).context("Failed to serialize tags")?,
+        })
+    }
 }
diff --git a/src/common/src/pipeline_tags.rs b/src/common/src/pipeline_tags.rs
index 2cc0a09f..f13f2340 100644
--- a/src/common/src/pipeline_tags.rs
+++ b/src/common/src/pipeline_tags.rs
@@ -22,6 +22,10 @@ pub struct PipelineTags {
     #[clap(long, default_value = "Oncology")]
     pub team: String,
 
+    /// Organization ID
+    #[clap(long)]
+    pub organization_id: Option<String>,
+
     /// Others: Any other tag you'd like to specify
     #[clap(long, value_delimiter = ',')]
     pub others: Vec<String>,
@@ -35,6 +39,7 @@ impl Default for PipelineTags {
             user_operator: "tracer_user".into(),
             department: "dev".into(),
             team: "dev".into(),
+            organization_id: None,
             others: vec![],
         }
     }
diff --git a/src/common/src/recorder.rs b/src/common/src/recorder.rs
index 5ff0cbd3..14eb7dfd 100644
--- a/src/common/src/recorder.rs
+++ b/src/common/src/recorder.rs
@@ -1,5 +1,5 @@
 use crate::event::attributes::EventAttributes;
-use crate::event::{Event, EventType, ProcessStatus, ProcessType};
+use crate::event::{Event, ProcessStatus};
 use crate::pipeline_tags::PipelineTags;
 use chrono::{DateTime, Utc};
 
@@ -45,23 +45,21 @@ impl EventRecorder {
     pub fn record_event(
         &mut self,
         process_status: ProcessStatus,
-        message: String,
+        body: String,
         attributes: Option<EventAttributes>,
         timestamp: Option<DateTime<Utc>>,
     ) {
-        let event = Event {
-            timestamp: timestamp.unwrap_or_else(Utc::now),
-            message,
-            event_type: EventType::ProcessStatus,
-            process_type: ProcessType::Pipeline,
-            process_status,
-            attributes,
-            // NOTE: not a fan of constant cloning so would look for an alt
-            run_name: self.run_name.clone(),
-            run_id: self.run_id.clone(),
-            pipeline_name: self.pipeline_name.clone(),
-            tags: self.tags.clone(),
-        };
+        let event = Event::builder()
+            .body(body)
+            .timestamp(timestamp.unwrap_or_else(Utc::now))
+            .process_status(process_status)
+            .pipeline_name(self.pipeline_name.clone())
+            .run_name(self.run_name.clone())
+            .run_id(self.run_id.clone())
+            .tags(self.tags.clone())
+            .attributes(attributes)
+            .build();
+
         self.events.push(event);
     }
 
@@ -92,17 +90,21 @@ impl Default for EventRecorder {
 
 #[cfg(test)]
 mod tests {
+    use crate::event::attributes::process::DataSetsProcessed;
+
     use super::*;
     use crate::event::attributes::EventAttributes;
-    use crate::event::ProcessStatus;
-    use serde_json::json;
+    use crate::event::{EventType, ProcessStatus, ProcessType};
 
     #[test]
     fn test_record_event() {
         let mut recorder = EventRecorder::default();
         let message = "[event_recorder.rs]Test event".to_string();
-        let attributes = Some(EventAttributes::Other(json!({"key": "value"})));
-
+        let attributes = Some(EventAttributes::ProcessDatasetStats(DataSetsProcessed {
+            datasets: "".to_string(),
+            total: 2,
+            trace_id: None,
+        }));
         recorder.record_event(
             ProcessStatus::ToolExecution,
             message.clone(),
@@ -113,13 +115,13 @@ mod tests {
         assert_eq!(recorder.len(), 1);
 
         let event = &recorder.get_events()[0];
-        assert_eq!(event.message, message);
+        assert_eq!(event.body, message);
         assert_eq!(event.event_type, EventType::ProcessStatus);
         assert_eq!(event.process_type, ProcessType::Pipeline);
         assert_eq!(event.process_status, ProcessStatus::ToolExecution);
         assert!(matches!(
             event.attributes.clone().unwrap(),
-            EventAttributes::Other(_)
+            EventAttributes::ProcessDatasetStats(_)
         ));
     }
 
@@ -140,17 +142,21 @@ mod tests {
 
     #[test]
     fn test_event_type_as_str() {
-        assert_eq!(ProcessStatus::FinishedRun.as_str(), "finished_run");
-        assert_eq!(ProcessStatus::ToolExecution.as_str(), "tool_execution");
-        assert_eq!(ProcessStatus::MetricEvent.as_str(), "metric_event");
-        assert_eq!(ProcessStatus::TestEvent.as_str(), "test_event");
+        assert_eq!(&ProcessStatus::FinishedRun.to_string(), "finished_run");
+        assert_eq!(&ProcessStatus::ToolExecution.to_string(), "tool_execution");
+        assert_eq!(&ProcessStatus::MetricEvent.to_string(), "metric_event");
+        assert_eq!(&ProcessStatus::TestEvent.to_string(), "test_event");
     }
 
     #[test]
     fn test_record_test_event() {
         let mut recorder = EventRecorder::default();
         let message = "Test event for testing".to_string();
-        let attributes = Some(EventAttributes::Other(json!({"test_key": "test_value"})));
+        let attributes = Some(EventAttributes::ProcessDatasetStats(DataSetsProcessed {
+            datasets: "".to_string(),
+            total: 2,
+            trace_id: None,
+        }));
 
         recorder.record_event(
             ProcessStatus::TestEvent,
@@ -162,13 +168,13 @@ mod tests {
         assert_eq!(recorder.len(), 1);
 
         let event = &recorder.get_events()[0];
-        assert_eq!(event.message, message);
+        assert_eq!(event.body, message);
         assert_eq!(event.event_type, EventType::ProcessStatus);
         assert_eq!(event.process_type, ProcessType::Pipeline);
         assert_eq!(event.process_status, ProcessStatus::TestEvent);
         assert!(matches!(
             event.attributes.clone().unwrap(),
-            EventAttributes::Other(_)
+            EventAttributes::ProcessDatasetStats(_)
         ));
     }
 }
diff --git a/src/daemon/src/daemon.rs b/src/daemon/src/daemon.rs
index 91222ec3..f82836d1 100644
--- a/src/daemon/src/daemon.rs
+++ b/src/daemon/src/daemon.rs
@@ -74,7 +74,6 @@ pub async fn monitor_processes_with_tracer_client(tracer_client: &mut TracerClie
     tracer_client.poll_process_metrics().await?;
     tracer_client.poll_syslog().await?;
     tracer_client.poll_stdout_stderr().await?;
-    tracer_client.poll_nextflow_log().await?;
     tracer_client.refresh_sysinfo();
     tracer_client.reset_just_started_process_flag();
     Ok(())
diff --git a/src/extracts/src/lib.rs b/src/extracts/src/lib.rs
index 0ba2c453..8f9a9508 100644
--- a/src/extracts/src/lib.rs
+++ b/src/extracts/src/lib.rs
@@ -6,4 +6,3 @@ pub mod stdout;
 pub mod syslog;
 
 mod data_samples;
-mod nextflow_log_watcher;
diff --git a/src/extracts/src/nextflow_log_watcher.rs b/src/extracts/src/nextflow_log_watcher.rs
deleted file mode 100644
index 41f83e21..00000000
--- a/src/extracts/src/nextflow_log_watcher.rs
+++ /dev/null
@@ -1,208 +0,0 @@
-use anyhow::Result;
-use chrono::Utc;
-use std::collections::HashMap;
-use std::path::{Path, PathBuf};
-use std::time::{Duration, Instant};
-use sysinfo::Pid;
-use tokio::fs::OpenOptions;
-use tokio::io::{AsyncBufReadExt, AsyncSeekExt, BufReader, SeekFrom};
-use tracer_common::event::attributes::system_metrics::NextflowLog;
-use tracer_common::event::attributes::EventAttributes;
-use tracer_common::event::ProcessStatus;
-use tracer_common::recorder::EventRecorder;
-
-/// ## Approach:
-/// - Limit search space of .nextflow.log files for possible locations: E:g $HOME, $PWD, maybe fallback to
-/// - Nextflow logs files are closer to the root directory and not deeper into the tree
-/// - Seperate Scanning and log processing. Scanning is a blocking task and should be treated as such
-/// - syncronization primitive to handle updating path when the log is found
-/// - Only poll when a path is found
-/// - Keep track of last processed point in file. And Seek from there on next poll cycle
-pub struct NextflowLogWatcher {
-    session_uuid: Option<String>,
-    jobs: Vec<String>,
-    processes: HashMap<Pid, PathBuf>, // Pid -> working_directory
-    last_poll_time: Option<Instant>,
-}
-
-impl Default for NextflowLogWatcher {
-    fn default() -> Self {
-        Self::new()
-    }
-}
-
-impl NextflowLogWatcher {
-    pub fn new() -> Self {
-        tracing::info!("Initializing new NextflowLogWatcher");
-        Self {
-            session_uuid: None,
-            jobs: Vec::new(),
-            processes: HashMap::new(),
-            last_poll_time: None,
-        }
-    }
-
-    fn reset_state(&mut self) {
-        tracing::info!("Resetting NextflowLogWatcher state - clearing session and jobs data");
-        self.session_uuid = None;
-        self.jobs.clear();
-    }
-
-    pub async fn poll_nextflow_log(&mut self, logs: &mut EventRecorder) -> Result<()> {
-        const POLL_INTERVAL: Duration = Duration::from_secs(10);
-
-        let now = Instant::now();
-        if let Some(last_poll) = self.last_poll_time {
-            if now.duration_since(last_poll) < POLL_INTERVAL {
-                return Ok(());
-            }
-        }
-
-        let start_time = tokio::time::Instant::now();
-        tracing::info!("Starting Nextflow log polling cycle");
-
-        let nextflow_logs_paths = self.processes.values().cloned().collect::<Vec<_>>();
-        tracing::info!(
-            "Found {} Nextflow log files to process",
-            nextflow_logs_paths.len()
-        );
-
-        for path in nextflow_logs_paths {
-            if path.exists() {
-                tracing::info!("Processing Nextflow log file: {:?}", path);
-                self.process_log_file(path.as_path(), logs).await?;
-            } else {
-                tracing::warn!("Nextflow log file not found: {:?}", path);
-            }
-        }
-
-        self.last_poll_time = Some(now);
-        tracing::info!(
-            "Completed Nextflow log polling cycle in {:?}",
-            start_time.elapsed()
-        );
-        Ok(())
-    }
-
-    async fn process_log_file(&mut self, log_path: &Path, logs: &mut EventRecorder) -> Result<()> {
-        let mut file = OpenOptions::new()
-            .read(true)
-            .open(log_path)
-            .await
-            .map_err(|err| {
-                tracing::error!("Error opening .nextflow.log at {:?}: {}", log_path, err);
-                anyhow::anyhow!("Error opening log file: {}", err)
-            })?;
-
-        file.seek(SeekFrom::Start(0)).await?;
-
-        self.reset_state();
-
-        let mut reader = BufReader::new(file);
-        let mut line = String::new();
-        let mut lines_processed = 0;
-        let mut session_found = false;
-        let mut jobs_found = 0;
-
-        while reader.read_line(&mut line).await? > 0 {
-            lines_processed += 1;
-            if line.contains("Session UUID:") {
-                session_found = true;
-            }
-            if line.contains("job=") {
-                jobs_found += 1;
-            }
-            self.process_log_line(&line);
-            line.clear();
-        }
-
-        tracing::info!(
-            "Processed Nextflow log file {:?}: {} lines, {} session(s), {} job(s)",
-            log_path,
-            lines_processed,
-            if session_found { 1 } else { 0 },
-            jobs_found
-        );
-
-        if let Some(session_uuid) = &self.session_uuid {
-            let message = format!(
-                "[CLI] Nextflow log event for session uuid: {} with {} jobs in file {:?}",
-                session_uuid,
-                self.jobs.len(),
-                log_path
-            );
-
-            let nextflow_log = EventAttributes::NextflowLog(NextflowLog {
-                session_uuid: Some(session_uuid.clone()),
-                jobs_ids: Some(self.jobs.clone()),
-            });
-
-            tracing::info!("Recording Nextflow log event: {}", message);
-            logs.record_event(
-                ProcessStatus::NextflowLogEvent,
-                message,
-                Some(nextflow_log),
-                Some(Utc::now()),
-            );
-        }
-
-        Ok(())
-    }
-
-    fn process_log_line(&mut self, line: &str) {
-        if line.contains("Session UUID:") {
-            if let Some(uuid) = extract_session_uuid(line) {
-                tracing::info!("Found new Nextflow session UUID: {}", uuid);
-                self.session_uuid = Some(uuid);
-            }
-        }
-
-        if line.contains("job=") {
-            if let Some(job_id) = extract_job_id(line) {
-                tracing::info!(
-                    "Found Nextflow job ID: {} for session: {:?}",
-                    job_id,
-                    self.session_uuid
-                );
-                self.jobs.push(job_id);
-            }
-        }
-    }
-
-    pub fn add_process(&mut self, pid: Pid, working_directory: PathBuf) {
-        tracing::info!(
-            "Adding new Nextflow process: pid={}, working_dir={:?}",
-            pid,
-            working_directory
-        );
-        self.processes.insert(pid, working_directory);
-    }
-
-    pub fn remove_process(&mut self, pid: Pid) {
-        if let Some(working_directory) = self.processes.remove(&pid) {
-            tracing::info!(
-                "Removing Nextflow process: pid={}, working_dir={:?}",
-                pid,
-                working_directory
-            );
-        }
-    }
-
-    pub fn get_process_working_directory(&self, pid: Pid) -> Option<&PathBuf> {
-        self.processes.get(&pid)
-    }
-}
-
-fn extract_session_uuid(line: &str) -> Option<String> {
-    line.split("Session UUID:")
-        .nth(1)
-        .map(|uuid_part| uuid_part.trim().to_string())
-}
-
-fn extract_job_id(line: &str) -> Option<String> {
-    if let Some(job_part) = line.split("job=").nth(1) {
-        Some(job_part.split(';').next()?.trim().to_string())
-    } else {
-        None
-    }
-}
diff --git a/src/extracts/src/process_watcher.rs b/src/extracts/src/process_watcher.rs
index 474d5d7e..99768186 100644
--- a/src/extracts/src/process_watcher.rs
+++ b/src/extracts/src/process_watcher.rs
@@ -1,19 +1,16 @@
 // src/process_watcher.rs
 
-use crate::nextflow_log_watcher::NextflowLogWatcher;
-
 use tracer_common::event::ProcessStatus as TracerProcessStatus;
 
 use crate::data_samples::DATA_SAMPLES_EXT;
 use crate::file_watcher::FileWatcher;
 use anyhow::Result;
 use chrono::{DateTime, Utc};
-use itertools::Itertools;
 use serde::Deserialize;
 use serde::Serialize;
 use std::collections::HashMap;
 use std::collections::{hash_map::Entry::Vacant, HashSet};
-use std::path::{Path, PathBuf};
+use std::path::Path;
 use std::time::Duration;
 use sysinfo::{Pid, Process, ProcessStatus, System};
 use tracer_common::event::attributes::process::{
@@ -27,9 +24,7 @@ pub struct ProcessWatcher {
     targets: Vec<Target>,
     seen: HashMap<Pid, Proc>,
     process_tree: HashMap<Pid, ProcessTreeNode>,
-    // We want to track unique data samples we come across when monitoring process args
-    datasamples_tracker: HashSet<String>,
-    nextflow_log_watcher: NextflowLogWatcher,
+    datasamples_tracker: HashMap<String, HashSet<String>>, // this hashmap groups datasets by the nextflow session uuid
 }
 
 enum ProcLastUpdate {
@@ -82,8 +77,7 @@ impl ProcessWatcher {
             targets,
             seen: HashMap::new(),
             process_tree: HashMap::new(),
-            datasamples_tracker: HashSet::new(),
-            nextflow_log_watcher: NextflowLogWatcher::new(),
+            datasamples_tracker: HashMap::new(),
         }
     }
 
@@ -192,10 +186,6 @@ impl ProcessWatcher {
         let mut to_remove = vec![];
         for (pid, proc) in self.seen.iter() {
             if !system.processes().contains_key(pid) {
-                // Check if this was a Nextflow process and notify the watcher
-                if proc.name.contains("nextflow") {
-                    self.nextflow_log_watcher.remove_process(*pid);
-                }
                 self.log_completed_process(pid, proc, event_logger)?;
                 to_remove.push(*pid);
             }
@@ -324,9 +314,10 @@ impl ProcessWatcher {
         Ok(())
     }
 
-    fn read_process_env(proc: &Process) -> (Option<String>, Option<String>) {
+    fn read_process_env(proc: &Process) -> (Option<String>, Option<String>, Option<String>) {
         let mut container_id = None;
         let mut job_id = None;
+        let mut trace_id = None;
 
         // Try to read environment variables
         for env_var in proc.environ() {
@@ -334,12 +325,13 @@ impl ProcessWatcher {
                 match key {
                     "AWS_BATCH_JOB_ID" => job_id = Some(value.to_string()),
                     "HOSTNAME" => container_id = Some(value.to_string()),
+                    "TRACER_TRACE_ID" => trace_id = Some(value.to_string()),
                     _ => continue,
                 }
             }
         }
 
-        (container_id, job_id)
+        (container_id, job_id, trace_id)
     }
 
     pub fn gather_process_data(
@@ -348,7 +340,7 @@ impl ProcessWatcher {
         display_name: Option<String>,
     ) -> ProcessProperties {
         let start_time = Utc::now();
-        let (container_id, job_id) = Self::read_process_env(proc);
+        let (container_id, job_id, trace_id) = Self::read_process_env(proc);
         let working_directory = proc.cwd().map(|p| p.to_string_lossy().to_string());
 
         ProcessProperties {
@@ -377,6 +369,7 @@ impl ProcessWatcher {
             container_id,
             job_id,
             working_directory,
+            trace_id,
         }
     }
 
@@ -443,6 +436,7 @@ impl ProcessWatcher {
                     container_id: None,
                     job_id: None,
                     working_directory: None,
+                    trace_id: None,
                 },
             }
         }
@@ -484,23 +478,6 @@ impl ProcessWatcher {
 
         let mut properties = Self::gather_process_data(&pid, p, Some(display_name.clone()));
 
-        // Check if this is a Nextflow process and notify the watcher
-        if display_name.contains("nextflow") {
-            tracing::info!(
-                "[{}] Found Nextflow process: pid={}, name={}, working_dir={:?}",
-                Utc::now(),
-                pid,
-                display_name,
-                properties.working_directory
-            );
-            if let Some(working_dir) = &properties.working_directory {
-                self.nextflow_log_watcher.add_process(
-                    pid,
-                    PathBuf::from(working_dir.clone()).join(".nextflow.log"),
-                );
-            }
-        }
-
         let cmd_arguments = p.cmd();
 
         let mut input_files = vec![];
@@ -540,11 +517,11 @@ impl ProcessWatcher {
         event_logger.record_event(
             TracerProcessStatus::ToolExecution,
             format!("[{}] Tool process: {}", start_time, &display_name),
-            Some(EventAttributes::Process(properties)),
+            Some(EventAttributes::Process(properties.clone())),
             None,
         );
 
-        self.log_datasets_in_process(event_logger, cmd_arguments);
+        self.log_datasets_in_process(event_logger, cmd_arguments, properties);
 
         Ok(())
     }
@@ -680,16 +657,36 @@ impl ProcessWatcher {
     }
 
     /// Logs the unique datasets processed or in process
-    fn log_datasets_in_process(&mut self, event_logger: &mut EventRecorder, cmd: &[String]) {
+    fn log_datasets_in_process(
+        &mut self,
+        event_logger: &mut EventRecorder,
+        cmd: &[String],
+        properties: ProcessProperties,
+    ) {
+        let trace_id: Option<String> = properties.trace_id.clone();
+
         for arg in cmd.iter() {
             if DATA_SAMPLES_EXT.iter().any(|ext| arg.ends_with(ext)) {
-                self.datasamples_tracker.insert(arg.clone());
+                self.datasamples_tracker
+                    .entry(trace_id.clone().unwrap_or_default())
+                    .or_default()
+                    .insert(arg.clone());
             }
         }
 
+        // TODO change this logic
         let properties = DataSetsProcessed {
-            datasets: self.datasamples_tracker.iter().join(", "),
-            total: self.datasamples_tracker.len() as u64,
+            datasets: self
+                .datasamples_tracker
+                .get(&trace_id.clone().unwrap_or_default())
+                .map(|set| set.iter().cloned().collect::<Vec<_>>().join(", "))
+                .unwrap_or_default(),
+            total: self
+                .datasamples_tracker
+                .get(&trace_id.clone().unwrap_or_default())
+                .unwrap_or(&HashSet::new())
+                .len() as u64,
+            trace_id,
         };
 
         event_logger.record_event(
@@ -725,14 +722,6 @@ impl ProcessWatcher {
     pub fn preview_targets_count(&self) -> usize {
         self.seen.len()
     }
-
-    pub fn get_nextflow_log_watcher(&self) -> &NextflowLogWatcher {
-        &self.nextflow_log_watcher
-    }
-
-    pub fn get_nextflow_log_watcher_mut(&mut self) -> &mut NextflowLogWatcher {
-        &mut self.nextflow_log_watcher
-    }
 }
 
 #[cfg(test)]
@@ -776,6 +765,7 @@ mod tests {
                 container_id: None,
                 job_id: None,
                 working_directory: None,
+                trace_id: None,
             };
 
             let node = ProcessTreeNode {
@@ -817,7 +807,7 @@ mod tests {
     }
 
     #[test]
-    fn test_count_dataset_matches_works() {
+    fn test_count_dataset_matches_works_without_trace_id() {
         let command: Vec<String> =
             "kallisto quant -t 4 -i control_index -o ./control_quant_9 control1_1.fa control1_2.fa"
                 .split(" ")
@@ -826,8 +816,18 @@ mod tests {
         let mut events_logger = EventRecorder::default();
 
         let mut process_watcher = ProcessWatcher::new(vec![]);
-        process_watcher.log_datasets_in_process(&mut events_logger, &command);
-        assert_eq!(process_watcher.datasamples_tracker.len(), 2);
+
+        let process_properties = create_process_properties_with_trace_id(None);
+
+        process_watcher.log_datasets_in_process(
+            &mut events_logger,
+            &command,
+            process_properties.clone(),
+        );
+        assert_eq!(
+            process_watcher.datasamples_tracker.get("").unwrap().len(),
+            2
+        );
 
         let command: Vec<String> =
             "kallisto quant -t 4 -i control_index -o ./control_quant_9 control1_3.fa control1_4.fa"
@@ -835,7 +835,78 @@ mod tests {
                 .map(String::from)
                 .collect();
 
-        process_watcher.log_datasets_in_process(&mut events_logger, &command);
-        assert_eq!(process_watcher.datasamples_tracker.len(), 4);
+        process_watcher.log_datasets_in_process(&mut events_logger, &command, process_properties);
+        assert_eq!(
+            process_watcher.datasamples_tracker.get("").unwrap().len(),
+            4
+        );
+    }
+
+    #[test]
+    fn test_count_dataset_matches_works_with_trace_id() {
+        let command: Vec<String> =
+            "kallisto quant -t 4 -i control_index -o ./control_quant_9 control1_1.fa control1_2.fa"
+                .split(" ")
+                .map(String::from)
+                .collect();
+        let mut events_logger = EventRecorder::default();
+
+        let mut process_watcher = ProcessWatcher::new(vec![]);
+        let process_properties =
+            create_process_properties_with_trace_id(Some("test_trace_id".to_string()));
+        process_watcher.log_datasets_in_process(
+            &mut events_logger,
+            &command,
+            process_properties.clone(),
+        );
+        assert_eq!(
+            process_watcher
+                .datasamples_tracker
+                .get("test_trace_id")
+                .unwrap()
+                .len(),
+            2
+        );
+
+        let command: Vec<String> =
+            "kallisto quant -t 4 -i control_index -o ./control_quant_9 control1_3.fa control1_4.fa"
+                .split(" ")
+                .map(String::from)
+                .collect();
+
+        process_watcher.log_datasets_in_process(&mut events_logger, &command, process_properties);
+        assert_eq!(
+            process_watcher
+                .datasamples_tracker
+                .get("test_trace_id")
+                .unwrap()
+                .len(),
+            4
+        );
+    }
+
+    fn create_process_properties_with_trace_id(trace_id: Option<String>) -> ProcessProperties {
+        ProcessProperties {
+            tool_name: "test".to_string(),
+            tool_pid: "test".to_string(),
+            tool_parent_pid: "test".to_string(),
+            tool_binary_path: "test".to_string(),
+            tool_cmd: "test".to_string(),
+            start_timestamp: "test".to_string(),
+            process_cpu_utilization: 5.5,
+            process_memory_usage: 5,
+            process_memory_virtual: 5,
+            process_run_time: 5,
+            process_disk_usage_read_last_interval: 5,
+            process_disk_usage_write_last_interval: 5,
+            process_disk_usage_read_total: 5,
+            process_disk_usage_write_total: 5,
+            process_status: "test".to_string(),
+            input_files: Some(Vec::new()),
+            container_id: Some("test".to_string()),
+            job_id: Some("test".to_string()),
+            working_directory: Some("test".to_string()),
+            trace_id,
+        }
     }
 }


Reviews:
user: login: blaginin
state: COMMENTED
author_association: NONE
submitted_at: 2025-04-15T19:43:14Z
user: login: blaginin
state: COMMENTED
author_association: NONE
submitted_at: 2025-04-15T19:47:35Z
user: login: Iamdavidonuh
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T19:48:16Z
user: login: blaginin
state: COMMENTED
author_association: NONE
submitted_at: 2025-04-15T19:49:07Z
user: login: blaginin
state: COMMENTED
author_association: NONE
submitted_at: 2025-04-15T19:50:38Z
user: login: Iamdavidonuh
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T00:52:53Z
user: login: Iamdavidonuh
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T00:54:14Z
user: login: MicheleVerriello
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T09:43:06Z
user: login: MicheleVerriello
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T09:47:53Z
user: login: MicheleVerriello
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T09:49:19Z
user: login: MicheleVerriello
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T09:51:48Z
user: login: MicheleVerriello
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T09:54:32Z
user: login: MicheleVerriello
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T10:02:40Z
user: login: MicheleVerriello
body: Don't see anything super critical, if you need to merge you can
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T10:03:28Z
user: login: blaginin
state: COMMENTED
author_association: NONE
submitted_at: 2025-04-16T10:46:45Z
user: login: blaginin
state: COMMENTED
author_association: NONE
submitted_at: 2025-04-16T10:49:22Z
user: login: blaginin
state: COMMENTED
author_association: NONE
submitted_at: 2025-04-16T10:54:52Z
user: login: blaginin
state: COMMENTED
author_association: NONE
submitted_at: 2025-04-16T10:58:03Z
user: login: blaginin
state: COMMENTED
author_association: NONE
submitted_at: 2025-04-16T10:59:13Z
user: login: blaginin
state: COMMENTED
author_association: NONE
submitted_at: 2025-04-16T11:04:36Z
user: login: blaginin
state: COMMENTED
author_association: NONE
submitted_at: 2025-04-16T11:07:44Z
user: login: blaginin
state: COMMENTED
author_association: NONE
submitted_at: 2025-04-16T11:10:08Z
user: login: blaginin
state: COMMENTED
author_association: NONE
submitted_at: 2025-04-16T11:10:58Z
user: login: blaginin
state: COMMENTED
author_association: NONE
submitted_at: 2025-04-16T11:12:35Z
user: login: blaginin
state: COMMENTED
author_association: NONE
submitted_at: 2025-04-16T11:17:58Z
user: login: blaginin
state: COMMENTED
author_association: NONE
submitted_at: 2025-04-16T11:18:42Z
user: login: blaginin
state: COMMENTED
author_association: NONE
submitted_at: 2025-04-16T11:19:36Z
user: login: MicheleVerriello
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T11:21:45Z
user: login: MicheleVerriello
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T11:25:22Z
user: login: blaginin
state: COMMENTED
author_association: NONE
submitted_at: 2025-04-16T11:29:25Z
user: login: blaginin
state: COMMENTED
author_association: NONE
submitted_at: 2025-04-16T11:30:26Z
user: login: blaginin
state: COMMENTED
author_association: NONE
submitted_at: 2025-04-16T11:37:44Z
user: login: Iamdavidonuh
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T11:49:53Z
user: login: blaginin
state: COMMENTED
author_association: NONE
submitted_at: 2025-04-16T11:51:48Z
user: login: Iamdavidonuh
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T11:52:08Z
user: login: Iamdavidonuh
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T11:52:55Z
user: login: blaginin
state: COMMENTED
author_association: NONE
submitted_at: 2025-04-16T11:53:06Z
user: login: blaginin
state: COMMENTED
author_association: NONE
submitted_at: 2025-04-16T11:59:05Z
user: login: blaginin
state: COMMENTED
author_association: NONE
submitted_at: 2025-04-16T12:15:46Z
user: login: MicheleVerriello
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T12:20:07Z
user: login: Iamdavidonuh
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T12:22:03Z
user: login: blaginin
state: COMMENTED
author_association: NONE
submitted_at: 2025-04-16T12:23:09Z
user: login: Iamdavidonuh
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T13:15:57Z
user: login: blaginin
state: COMMENTED
author_association: NONE
submitted_at: 2025-04-16T13:29:58Z
user: login: blaginin
state: COMMENTED
author_association: NONE
submitted_at: 2025-04-17T11:03:44Z
user: login: Iamdavidonuh
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T11:41:38Z
user: login: blaginin
state: APPROVED
author_association: NONE
submitted_at: 2025-04-17T14:24:34Z
user: login: blaginin
state: COMMENTED
author_association: NONE
submitted_at: 2025-04-17T14:25:07Z
user: login: blaginin
state: COMMENTED
author_association: NONE
submitted_at: 2025-04-17T14:25:33Z
user: login: Iamdavidonuh
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T14:41:22Z
user: login: MicheleVerriello
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T14:58:05Z

Review comments:
diff_hunk: @@ -0,0 +1,173 @@
+use chrono::{DateTime, Utc};
+use serde::{Deserialize, Serialize};
+use serde_json::{json, Value};
+
+use crate::exporters::db::EventInsert;
+use crate::types::cli::PipelineTags;
+use crate::types::event::{attributes::EventAttributes, Event, ProcessStatus};
+
+#[derive(Serialize, Deserialize, Debug, Clone)]
+pub struct OtelLog {
+    pub timestamp: DateTime<Utc>,
+    pub body: String,
+    pub severity_text: Option<String>,
+    pub severity_number: Option<u8>,
+    pub trace_id: Option<String>,
+    pub span_id: Option<String>,
+
+    pub source_type: String,
+    pub instrumentation_version: Option<String>,
+    pub instrumentation_type: Option<String>,
+
+    pub environment: Option<String>,
+    pub pipeline_type: Option<String>,
+    pub user_operator: Option<String>,
+    pub organization_id: Option<String>,
+    pub department: Option<String>,
+
+    pub run_id: Option<String>,
+    pub run_name: Option<String>,
+    pub pipeline_name: Option<String>,
+
+    pub job_id: Option<String>,
+    pub parent_job_id: Option<String>,
+    pub child_job_ids: Option<Vec<String>>,
+    pub workflow_engine: Option<String>,
+
+    pub ec2_cost_per_hour: Option<f64>,
+    pub cpu_usage: Option<f32>,
+    pub mem_used: Option<f64>,
+    pub processed_dataset: Option<i32>,
+    pub process_status: ProcessStatus,
+
+    pub attributes: Option<Value>,
+    pub resource_attributes: Option<Value>,
+    pub tags: Option<PipelineTags>,
+}
+
+impl From<Event> for OtelLog {
path: src/types/event/otel.rs
user: login: blaginin
body: for me to better understand - why do we need `OtelLog` at all? why can't it be `Event` itself? 
created_at: 2025-04-15T19:43:14Z
author_association: NONE
diff_hunk: @@ -162,37 +157,54 @@ impl AuroraClient {
         run_name: &str,
         run_id: &str,
         pipeline_name: &str,
-        data: impl IntoIterator<Item = &Event>,
+        logs: impl IntoIterator<Item = OtelLog>,
     ) -> Result<()> {
-        let now = Instant::now();
+        let now = std::time::Instant::now();
+        const PARAMS: usize = 29;
 
         const QUERY: &str = "INSERT INTO batch_jobs_logs (
-            data, job_id, run_name, run_id, pipeline_name, nextflow_session_uuid, job_ids,
-            tags, event_timestamp, ec2_cost_per_hour, cpu_usage, mem_used, processed_dataset,
-            process_status
-            )"; // when updating query, also update params
-        const PARAMS: usize = 13;
+                timestamp, body, severity_text, severity_number,
+                trace_id, span_id,
+                source_type, instrumentation_version, instrumentation_type,
+                environment, pipeline_type, user_operator, organization_id, department,
+                run_id, run_name, pipeline_name,
+                job_id, parent_job_id, child_job_ids, workflow_engine,
+                ec2_cost_per_hour, cpu_usage, mem_used, processed_dataset,
+                process_status,
+                attributes, resource_attributes, tags
+            )";
 
         fn _push_tuple(mut b: Separated<Postgres, &str>, event: EventInsert) {
-            b.push_bind(event.json_data)
-                .push_bind(event.job_id)
-                .push_bind(event.run_name)
+            b.push_bind(event.event_timestamp.naive_utc())
+                .push_bind(event.body)
+                .push_bind(event.severity_text)
+                .push_bind(event.severity_number)
+                .push_bind(event.trace_id.or_else(|| Some(event.run_id.clone())))
+                .push_bind(event.span_id)
+                .push_bind(event.source_type)
+                .push_bind(event.instrumentation_version)
+                .push_bind(event.instrumentation_type)
+                .push_bind(event.environment)
+                .push_bind(event.pipeline_type)
+                .push_bind(event.user_operator)
+                .push_bind(event.organization_id)
+                .push_bind(event.department)
                 .push_bind(event.run_id)
+                .push_bind(event.run_name)
                 .push_bind(event.pipeline_name)
-                .push_bind(event.nextflow_session_uuid)
-                .push_bind(event.job_ids)
-                .push_bind(event.tags_json)
-                .push_bind(event.event_timestamp.naive_utc())
+                .push_bind(event.job_id)
+                .push_bind(event.parent_job_id)
+                .push_bind(event.child_job_ids)
+                .push_bind(event.workflow_engine)
                 .push_bind(event.ec2_cost_per_hour)
                 .push_bind(event.cpu_usage)
                 .push_bind(event.mem_used)
                 .push_bind(event.processed_dataset)
-                .push_bind(event.process_status);
+                .push_bind(event.process_status)
+                .push_bind(event.attributes)
+                .push_bind(event.resource_attributes)
+                .push_bind(event.tags);
         }
-        // there's an alternative, much more efficient way to push values
path: src/exporters/db.rs
user: login: blaginin
body: is there a reson to remove this note? 😀 i think it's good to keep explanations here and there so we will not forget the context in the future
created_at: 2025-04-15T19:47:31Z
author_association: NONE
diff_hunk: @@ -162,37 +157,54 @@ impl AuroraClient {
         run_name: &str,
         run_id: &str,
         pipeline_name: &str,
-        data: impl IntoIterator<Item = &Event>,
+        logs: impl IntoIterator<Item = OtelLog>,
     ) -> Result<()> {
-        let now = Instant::now();
+        let now = std::time::Instant::now();
+        const PARAMS: usize = 29;
 
         const QUERY: &str = "INSERT INTO batch_jobs_logs (
-            data, job_id, run_name, run_id, pipeline_name, nextflow_session_uuid, job_ids,
-            tags, event_timestamp, ec2_cost_per_hour, cpu_usage, mem_used, processed_dataset,
-            process_status
-            )"; // when updating query, also update params
-        const PARAMS: usize = 13;
+                timestamp, body, severity_text, severity_number,
+                trace_id, span_id,
+                source_type, instrumentation_version, instrumentation_type,
+                environment, pipeline_type, user_operator, organization_id, department,
+                run_id, run_name, pipeline_name,
+                job_id, parent_job_id, child_job_ids, workflow_engine,
+                ec2_cost_per_hour, cpu_usage, mem_used, processed_dataset,
+                process_status,
+                attributes, resource_attributes, tags
+            )";
 
         fn _push_tuple(mut b: Separated<Postgres, &str>, event: EventInsert) {
-            b.push_bind(event.json_data)
-                .push_bind(event.job_id)
-                .push_bind(event.run_name)
+            b.push_bind(event.event_timestamp.naive_utc())
+                .push_bind(event.body)
+                .push_bind(event.severity_text)
+                .push_bind(event.severity_number)
+                .push_bind(event.trace_id.or_else(|| Some(event.run_id.clone())))
+                .push_bind(event.span_id)
+                .push_bind(event.source_type)
+                .push_bind(event.instrumentation_version)
+                .push_bind(event.instrumentation_type)
+                .push_bind(event.environment)
+                .push_bind(event.pipeline_type)
+                .push_bind(event.user_operator)
+                .push_bind(event.organization_id)
+                .push_bind(event.department)
                 .push_bind(event.run_id)
+                .push_bind(event.run_name)
                 .push_bind(event.pipeline_name)
-                .push_bind(event.nextflow_session_uuid)
-                .push_bind(event.job_ids)
-                .push_bind(event.tags_json)
-                .push_bind(event.event_timestamp.naive_utc())
+                .push_bind(event.job_id)
+                .push_bind(event.parent_job_id)
+                .push_bind(event.child_job_ids)
+                .push_bind(event.workflow_engine)
                 .push_bind(event.ec2_cost_per_hour)
                 .push_bind(event.cpu_usage)
                 .push_bind(event.mem_used)
                 .push_bind(event.processed_dataset)
-                .push_bind(event.process_status);
+                .push_bind(event.process_status)
+                .push_bind(event.attributes)
+                .push_bind(event.resource_attributes)
+                .push_bind(event.tags);
         }
-        // there's an alternative, much more efficient way to push values
path: src/exporters/db.rs
user: login: Iamdavidonuh
body: Yeah Yeah, that was a mistake
created_at: 2025-04-15T19:48:15Z
author_association: CONTRIBUTOR
diff_hunk: @@ -162,37 +157,54 @@ impl AuroraClient {
         run_name: &str,
         run_id: &str,
         pipeline_name: &str,
-        data: impl IntoIterator<Item = &Event>,
+        logs: impl IntoIterator<Item = OtelLog>,
     ) -> Result<()> {
-        let now = Instant::now();
+        let now = std::time::Instant::now();
+        const PARAMS: usize = 29;
 
         const QUERY: &str = "INSERT INTO batch_jobs_logs (
-            data, job_id, run_name, run_id, pipeline_name, nextflow_session_uuid, job_ids,
-            tags, event_timestamp, ec2_cost_per_hour, cpu_usage, mem_used, processed_dataset,
-            process_status
-            )"; // when updating query, also update params
-        const PARAMS: usize = 13;
path: src/exporters/db.rs
user: login: blaginin
body: can params please be close to the query? and can we also keep the note?
created_at: 2025-04-15T19:49:07Z
author_association: NONE
diff_hunk: @@ -162,37 +157,54 @@ impl AuroraClient {
         run_name: &str,
         run_id: &str,
         pipeline_name: &str,
-        data: impl IntoIterator<Item = &Event>,
+        logs: impl IntoIterator<Item = OtelLog>,
     ) -> Result<()> {
-        let now = Instant::now();
+        let now = std::time::Instant::now();
+        const PARAMS: usize = 29;
 
         const QUERY: &str = "INSERT INTO batch_jobs_logs (
-            data, job_id, run_name, run_id, pipeline_name, nextflow_session_uuid, job_ids,
-            tags, event_timestamp, ec2_cost_per_hour, cpu_usage, mem_used, processed_dataset,
-            process_status
-            )"; // when updating query, also update params
-        const PARAMS: usize = 13;
+                timestamp, body, severity_text, severity_number,
+                trace_id, span_id,
+                source_type, instrumentation_version, instrumentation_type,
+                environment, pipeline_type, user_operator, organization_id, department,
+                run_id, run_name, pipeline_name,
+                job_id, parent_job_id, child_job_ids, workflow_engine,
+                ec2_cost_per_hour, cpu_usage, mem_used, processed_dataset,
+                process_status,
+                attributes, resource_attributes, tags
+            )";
 
         fn _push_tuple(mut b: Separated<Postgres, &str>, event: EventInsert) {
-            b.push_bind(event.json_data)
-                .push_bind(event.job_id)
-                .push_bind(event.run_name)
+            b.push_bind(event.event_timestamp.naive_utc())
+                .push_bind(event.body)
+                .push_bind(event.severity_text)
+                .push_bind(event.severity_number)
+                .push_bind(event.trace_id.or_else(|| Some(event.run_id.clone())))
path: src/exporters/db.rs
user: login: blaginin
body: trace is is trace_id or run_id? 😀 this feels a bit counterintuitive - why can't we just push `null` and later handle this on the query side?
created_at: 2025-04-15T19:50:38Z
author_association: NONE
diff_hunk: @@ -0,0 +1,173 @@
+use chrono::{DateTime, Utc};
+use serde::{Deserialize, Serialize};
+use serde_json::{json, Value};
+
+use crate::exporters::db::EventInsert;
+use crate::types::cli::PipelineTags;
+use crate::types::event::{attributes::EventAttributes, Event, ProcessStatus};
+
+#[derive(Serialize, Deserialize, Debug, Clone)]
+pub struct OtelLog {
+    pub timestamp: DateTime<Utc>,
+    pub body: String,
+    pub severity_text: Option<String>,
+    pub severity_number: Option<u8>,
+    pub trace_id: Option<String>,
+    pub span_id: Option<String>,
+
+    pub source_type: String,
+    pub instrumentation_version: Option<String>,
+    pub instrumentation_type: Option<String>,
+
+    pub environment: Option<String>,
+    pub pipeline_type: Option<String>,
+    pub user_operator: Option<String>,
+    pub organization_id: Option<String>,
+    pub department: Option<String>,
+
+    pub run_id: Option<String>,
+    pub run_name: Option<String>,
+    pub pipeline_name: Option<String>,
+
+    pub job_id: Option<String>,
+    pub parent_job_id: Option<String>,
+    pub child_job_ids: Option<Vec<String>>,
+    pub workflow_engine: Option<String>,
+
+    pub ec2_cost_per_hour: Option<f64>,
+    pub cpu_usage: Option<f32>,
+    pub mem_used: Option<f64>,
+    pub processed_dataset: Option<i32>,
+    pub process_status: ProcessStatus,
+
+    pub attributes: Option<Value>,
+    pub resource_attributes: Option<Value>,
+    pub tags: Option<PipelineTags>,
+}
+
+impl From<Event> for OtelLog {
path: src/types/event/otel.rs
user: login: Iamdavidonuh
body: No reason at all.. It's used as a test for insertion. 
PS: keep in mind this is still in draft 
created_at: 2025-04-16T00:52:53Z
author_association: CONTRIBUTOR
diff_hunk: @@ -162,37 +157,54 @@ impl AuroraClient {
         run_name: &str,
         run_id: &str,
         pipeline_name: &str,
-        data: impl IntoIterator<Item = &Event>,
+        logs: impl IntoIterator<Item = OtelLog>,
     ) -> Result<()> {
-        let now = Instant::now();
+        let now = std::time::Instant::now();
+        const PARAMS: usize = 29;
 
         const QUERY: &str = "INSERT INTO batch_jobs_logs (
-            data, job_id, run_name, run_id, pipeline_name, nextflow_session_uuid, job_ids,
-            tags, event_timestamp, ec2_cost_per_hour, cpu_usage, mem_used, processed_dataset,
-            process_status
-            )"; // when updating query, also update params
-        const PARAMS: usize = 13;
+                timestamp, body, severity_text, severity_number,
+                trace_id, span_id,
+                source_type, instrumentation_version, instrumentation_type,
+                environment, pipeline_type, user_operator, organization_id, department,
+                run_id, run_name, pipeline_name,
+                job_id, parent_job_id, child_job_ids, workflow_engine,
+                ec2_cost_per_hour, cpu_usage, mem_used, processed_dataset,
+                process_status,
+                attributes, resource_attributes, tags
+            )";
 
         fn _push_tuple(mut b: Separated<Postgres, &str>, event: EventInsert) {
-            b.push_bind(event.json_data)
-                .push_bind(event.job_id)
-                .push_bind(event.run_name)
+            b.push_bind(event.event_timestamp.naive_utc())
+                .push_bind(event.body)
+                .push_bind(event.severity_text)
+                .push_bind(event.severity_number)
+                .push_bind(event.trace_id.or_else(|| Some(event.run_id.clone())))
path: src/exporters/db.rs
user: login: Iamdavidonuh
body: So it can be null but the idea is to have the run_id be the trace_id to be able to build a parent -> child relationship which `aws_batch nextflow` where one job can be spawned on multiple machines
created_at: 2025-04-16T00:54:13Z
author_association: CONTRIBUTOR
diff_hunk: @@ -41,11 +41,7 @@ Then do:
 
 ```bash
 export DATABASE_URL=tracer-cluster-v2-instance-1.cdgizpzxtdp6.us-east-1.rds.amazonaws.com:5432/tracer_db
-<<<<<<< HEAD
 ```
 
 ## Running A Local Environment Test
 init --pipeline-name rnaseq-demo-123 --environment demo --user-operator vincent --pipeline-type rnaseq
-=======
path: docs/DEVELOPMENT.md
user: login: MicheleVerriello
body: here we have some confilcts, probably during some merge
created_at: 2025-04-16T09:43:05Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,29 @@
+-- Drop otel_logs table
+DROP TABLE IF EXISTS batch_jobs_logs;
+
+-- Recreate batch_jobs_logs as it existed before
+CREATE TABLE IF NOT EXISTS batch_jobs_logs (
+    id SERIAL PRIMARY KEY,
+    data JSONB NOT NULL,
+    job_id TEXT,
+    creation_date TIMESTAMP DEFAULT NOW(),
+    run_name TEXT,
+    run_id TEXT,
+    pipeline_name TEXT,
+    nextflow_session_uuid TEXT,
+    job_ids TEXT[],
+    tags JSONB,
+    event_timestamp TIMESTAMP,
+    ec2_cost_per_hour FLOAT,
+    cpu_usage FLOAT,
+    mem_used FLOAT,
+    processed_dataset INT,
+    process_status TEXT
+);
+
+-- Rebuild indexes if needed (can be omitted if you don’t need rollback performance)
+CREATE INDEX IF NOT EXISTS idx_batch_jobs_logs_metrics
+    ON batch_jobs_logs (job_id, pipeline_name, tags, event_timestamp, ec2_cost_per_hour, cpu_usage, mem_used, processed_dataset);
+
+CREATE INDEX IF NOT EXISTS idx_batch_jobs_logs_process_status
+    ON batch_jobs_logs (process_status);
path: migrations/20250414175511_new_schema_update.down.sql
user: login: MicheleVerriello
body: after the last index add `ANALYZE batch_jobs_logs;` to refresh the indexes
created_at: 2025-04-16T09:47:53Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,62 @@
+-- Drop the old table
+DROP TABLE IF EXISTS batch_jobs_logs;
+
+-- Create the new batch_jobs_logs table
+CREATE TABLE IF NOT EXISTS batch_jobs_logs (
+    event_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+
+    -- Core OTel fields
+    timestamp TIMESTAMPTZ NOT NULL,
+    body TEXT NOT NULL,
+    severity_text TEXT,
+    severity_number SMALLINT,
+    trace_id TEXT,
+    span_id TEXT,
+
+    -- Instrumentation metadata
+    source_type TEXT,
+    instrumentation_version TEXT,
+    instrumentation_type TEXT,
+
+    -- Tags (flattened from JSON)
+    environment TEXT,
+    pipeline_type TEXT,
+    user_operator TEXT,
+    department TEXT,
+    organization_id TEXT,
+
+    -- Pipeline context 
+    run_id TEXT,
+    run_name TEXT,
+    pipeline_name TEXT,
+
+    -- Execution tracing (generalized)
+    job_id TEXT,
+    parent_job_id TEXT,
+    child_job_ids TEXT[],
+    workflow_engine TEXT,
+
+    -- Performance/metrics fields
+    ec2_cost_per_hour FLOAT,
+    cpu_usage FLOAT,
+    mem_used FLOAT,
+    processed_dataset INT,
+    process_status TEXT,
+
+    -- Full structured logs and metadata
+    attributes JSONB,
+    resource_attributes JSONB,
+    tags JSONB
+);
+
+-- Indexes
+CREATE INDEX IF NOT EXISTS idx_batch_jobs_logs_timestamp ON batch_jobs_logs (timestamp);
+CREATE INDEX IF NOT EXISTS idx_batch_jobs_logs_run_pipeline ON batch_jobs_logs (run_id, pipeline_name);
+CREATE INDEX IF NOT EXISTS idx_batch_jobs_logs_job_trace ON batch_jobs_logs (job_id, parent_job_id, workflow_engine);
+CREATE INDEX IF NOT EXISTS idx_batch_jobs_logs_metrics ON batch_jobs_logs (
+    cpu_usage,
+    mem_used,
+    ec2_cost_per_hour,
+    processed_dataset
+);
+CREATE INDEX IF NOT EXISTS idx_batch_jobs_logs_process_status ON batch_jobs_logs (process_status);
path: migrations/20250414175511_new_schema_update.up.sql
user: login: MicheleVerriello
body: after the last index add `ANALYZE batch_jobs_logs;` to refresh the indexes
created_at: 2025-04-16T09:49:19Z
author_association: CONTRIBUTOR
diff_hunk: @@ -163,88 +74,89 @@ impl AuroraClient {
         run_name: &str,
         run_id: &str,
         pipeline_name: &str,
-        data: impl IntoIterator<Item = &Event>,
+        logs: impl IntoIterator<Item = Event>,
     ) -> Result<()> {
-        let now = Instant::now();
+        let now = std::time::Instant::now();
 
         const QUERY: &str = "INSERT INTO batch_jobs_logs (
-            data, job_id, run_name, run_id, pipeline_name, nextflow_session_uuid, job_ids,
-            tags, event_timestamp, ec2_cost_per_hour, cpu_usage, mem_used, processed_dataset,
-            process_status
-            )"; // when updating query, also update params
-        const PARAMS: usize = 13;
+                timestamp, body, severity_text, severity_number,
+                trace_id, span_id,
+                source_type, instrumentation_version, instrumentation_type,
+                environment, pipeline_type, user_operator, organization_id, department,
+                run_id, run_name, pipeline_name,
+                job_id, parent_job_id, child_job_ids, workflow_engine,
+                ec2_cost_per_hour, cpu_usage, mem_used, processed_dataset,
+                process_status,
+                attributes, resource_attributes, tags
+            )";
+        // when updating query, also update params
+        const PARAMS: usize = 29;
 
         fn _push_tuple(mut b: Separated<Postgres, &str>, event: EventInsert) {
-            b.push_bind(event.json_data)
-                .push_bind(event.job_id)
-                .push_bind(event.run_name)
+            b.push_bind(event.event_timestamp.naive_utc())
path: src/client/src/exporters/db.rs
user: login: MicheleVerriello
body: @blaginin @Iamdavidonuh just curious, is there in rust a way to push these binds in an more explicative way?
for example `b.push_bind("field_name", event.field_value)`
created_at: 2025-04-16T09:51:48Z
author_association: CONTRIBUTOR
diff_hunk: @@ -163,88 +74,89 @@ impl AuroraClient {
         run_name: &str,
         run_id: &str,
         pipeline_name: &str,
-        data: impl IntoIterator<Item = &Event>,
+        logs: impl IntoIterator<Item = Event>,
     ) -> Result<()> {
-        let now = Instant::now();
+        let now = std::time::Instant::now();
 
         const QUERY: &str = "INSERT INTO batch_jobs_logs (
-            data, job_id, run_name, run_id, pipeline_name, nextflow_session_uuid, job_ids,
-            tags, event_timestamp, ec2_cost_per_hour, cpu_usage, mem_used, processed_dataset,
-            process_status
-            )"; // when updating query, also update params
-        const PARAMS: usize = 13;
+                timestamp, body, severity_text, severity_number,
+                trace_id, span_id,
+                source_type, instrumentation_version, instrumentation_type,
+                environment, pipeline_type, user_operator, organization_id, department,
+                run_id, run_name, pipeline_name,
+                job_id, parent_job_id, child_job_ids, workflow_engine,
+                ec2_cost_per_hour, cpu_usage, mem_used, processed_dataset,
+                process_status,
+                attributes, resource_attributes, tags
+            )";
+        // when updating query, also update params
+        const PARAMS: usize = 29;
 
         fn _push_tuple(mut b: Separated<Postgres, &str>, event: EventInsert) {
-            b.push_bind(event.json_data)
-                .push_bind(event.job_id)
-                .push_bind(event.run_name)
+            b.push_bind(event.event_timestamp.naive_utc())
+                .push_bind(event.body)
+                .push_bind(event.severity_text)
+                .push_bind(event.severity_number)
+                .push_bind(event.trace_id.or_else(|| Some(event.run_id.clone())))
+                .push_bind(event.span_id)
+                .push_bind(event.source_type)
+                .push_bind(event.instrumentation_version)
+                .push_bind(event.instrumentation_type)
+                .push_bind(event.environment)
+                .push_bind(event.pipeline_type)
+                .push_bind(event.user_operator)
+                .push_bind(event.organization_id)
+                .push_bind(event.department)
                 .push_bind(event.run_id)
+                .push_bind(event.run_name)
                 .push_bind(event.pipeline_name)
-                .push_bind(event.nextflow_session_uuid)
-                .push_bind(event.job_ids)
-                .push_bind(event.tags_json)
-                .push_bind(event.event_timestamp.naive_utc())
+                .push_bind(event.job_id)
+                .push_bind(event.parent_job_id)
+                .push_bind(event.child_job_ids)
+                .push_bind(event.workflow_engine)
                 .push_bind(event.ec2_cost_per_hour)
                 .push_bind(event.cpu_usage)
                 .push_bind(event.mem_used)
                 .push_bind(event.processed_dataset)
-                .push_bind(event.process_status);
+                .push_bind(event.process_status)
+                .push_bind(event.attributes)
+                .push_bind(event.resource_attributes)
+                .push_bind(event.tags);
         }
         // there's an alternative, much more efficient way to push values
         // https://github.com/launchbadge/sqlx/blob/main/FAQ.md#how-can-i-bind-an-array-to-a-values-clause-how-can-i-do-bulk-inserts
         // however, unnest builds a tmp table from the array - and we're passing job_ids. Unnesting Arrays inside the arrays are tricky.
         // see https://github.com/launchbadge/sqlx/issues/1945
 
         info!(
-            "Inserting row for run_name: {}, pipeline_name: {}",
-            run_name, pipeline_name
+            "Inserting row for run_name: {}, pipeline_name: {}, run_id: {}",
+            run_name, pipeline_name, run_id
         );
 
         let mut builder = QueryBuilder::new(QUERY);
 
-        let mut data: Vec<_> = data
-            .into_iter()
-            .map(|e| {
-                EventInsert::try_new(
-                    e,
-                    run_name.to_string(),
-                    run_id.to_string(),
-                    pipeline_name.to_string(),
-                    e.process_status.as_str().to_string(),
-                )
-            })
-            .collect::<Result<Vec<_>>>()?;
+        let mut data: Vec<_> = logs.into_iter().map(EventInsert::from).collect();
 
         let rows_affected = match data.len() {
             0 => {
                 debug!("No data to insert");
                 return Ok(());
             }
-
             x if x * PARAMS >= BIND_LIMIT => {
-                debug!("Using transaction to insert data");
-                let mut transaction = self.pool.begin().await?;
-
+                debug!("Chunked insert with transaction due to bind limit");
+                let mut tx = self.pool.begin().await?;
path: src/client/src/exporters/db.rs
user: login: MicheleVerriello
body: not to do now, but let's try to keep the variable names readable like `transaction` instead of `tx`
created_at: 2025-04-16T09:54:32Z
author_association: CONTRIBUTOR
diff_hunk: @@ -79,7 +79,7 @@ jobs:
           TRACER_DATABASE_NAME: ${{ secrets.TRACER_DATABASE_NAME }}
           TRACER_DATABASE_HOST: ${{ secrets.TRACER_DATABASE_HOST }}
           DATABASE_URL: "postgresql://postgres:postgres@localhost:${{ job.services.postgres.ports[5432] }}/tracer_db" # localhost since VM
-        run: make test-tracer
+        run: ./scripts/migrate.sh && make test-tracer
path: .github/workflows/build-and-test.yml
user: login: MicheleVerriello
body: not sure about this, why do we have `TRACER_DATABASE_NAME` and `TRACER_DATABASE_HOST` that are pointing to prod, but then the `DATABASE_URL` is pointing to localhost?
created_at: 2025-04-16T10:02:40Z
author_association: CONTRIBUTOR
diff_hunk: @@ -79,7 +79,7 @@ jobs:
           TRACER_DATABASE_NAME: ${{ secrets.TRACER_DATABASE_NAME }}
           TRACER_DATABASE_HOST: ${{ secrets.TRACER_DATABASE_HOST }}
           DATABASE_URL: "postgresql://postgres:postgres@localhost:${{ job.services.postgres.ports[5432] }}/tracer_db" # localhost since VM
-        run: make test-tracer
+        run: ./scripts/migrate.sh && make test-tracer
path: .github/workflows/build-and-test.yml
user: login: blaginin
body: database url is pointing to localhost for cli tests - that's what i wrote here: https://github.com/Tracer-Cloud/tracer-client/pull/119#discussion_r2044546544


I do not think we should use prod in here, because if you open a PR which adds bad migrations, prod will be fucked. I believe `DATABASE_URL` `TRACER_DATABASE_HOST` and `TRACER_DATABASE_NAME` should also be localhost. Didn't change it in my PR not to increase it's size but maybe it's time to do that now? 
created_at: 2025-04-16T10:46:45Z
author_association: NONE
diff_hunk: @@ -17,4 +17,7 @@ serde = { workspace = true }
 serde_json = { workspace = true }
 tokio = { workspace = true }
 url = { workspace = true }
-tracer_aws = { workspace = true }
\ No newline at end of file
+tracer_aws = { workspace = true }
+uuid = { workspace = true }
+
+typed-builder = "0.18"
path: src/common/Cargo.toml
user: login: blaginin
body: let's add it to the workspace, so it'll be a single place to manage dependencies. here you can use `= { workspace = true }`
created_at: 2025-04-16T10:49:21Z
author_association: NONE
diff_hunk: @@ -36,37 +39,181 @@ pub enum ProcessStatus {
     NextflowLogEvent,
 }
 
-impl ProcessStatus {
-    pub fn as_str(&self) -> &'static str {
+impl std::fmt::Display for ProcessStatus {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
         match self {
-            ProcessStatus::NewRun => "new_run",
-            ProcessStatus::FinishedRun => "finished_run",
-            ProcessStatus::ToolExecution => "tool_execution",
-            ProcessStatus::FinishedToolExecution => "finished_tool_execution",
-            ProcessStatus::MetricEvent => "metric_event",
-            ProcessStatus::SyslogEvent => "syslog_event",
-            ProcessStatus::ToolMetricEvent => "tool_metric_event",
-            ProcessStatus::TestEvent => "test_event", // Handle TestEvent
-            ProcessStatus::RunStatusMessage => "run_status_message",
-            ProcessStatus::Alert => "alert",
-            ProcessStatus::DataSamplesEvent => "datasets_in_process",
-            ProcessStatus::NextflowLogEvent => "nextflow_log_event",
+            ProcessStatus::NewRun => write!(f, "new_run"),
+            ProcessStatus::FinishedRun => write!(f, "finished_run"),
+            ProcessStatus::ToolExecution => write!(f, "tool_execution"),
+            ProcessStatus::FinishedToolExecution => write!(f, "finished_tool_execution"),
+            ProcessStatus::ToolMetricEvent => write!(f, "tool_metric_event"),
+            ProcessStatus::MetricEvent => write!(f, "metric_event"),
+            ProcessStatus::SyslogEvent => write!(f, "syslog_event"),
+            ProcessStatus::RunStatusMessage => write!(f, "run_status_message"),
+            ProcessStatus::Alert => write!(f, "alert"),
+            ProcessStatus::DataSamplesEvent => write!(f, "datasets_in_process"),
+            ProcessStatus::TestEvent => write!(f, "test_event"),
+            ProcessStatus::NextflowLogEvent => write!(f, "nextflow_log_event"),
         }
     }
 }
-#[derive(serde::Serialize, serde::Deserialize, Debug, Clone)]
+
+fn default_span_id() -> Option<String> {
+    Some(Uuid::new_v4().to_string())
+}
+
+#[derive(Serialize, Deserialize, Debug, Clone, TypedBuilder)]
+#[builder(field_defaults(default))]
 pub struct Event {
     #[serde(with = "ts_seconds")]
     pub timestamp: DateTime<Utc>,
-    pub message: String,
 
+    #[builder(setter(into))]
+    pub body: String,
+
+    #[builder(default = EventType::ProcessStatus)]
     pub event_type: EventType,
+
+    #[builder(default = ProcessType::Pipeline)]
     pub process_type: ProcessType,
+
+    #[builder(default = ProcessStatus::TestEvent)]
     pub process_status: ProcessStatus,
 
     pub pipeline_name: Option<String>,
     pub run_name: Option<String>,
     pub run_id: Option<String>,
     pub attributes: Option<EventAttributes>,
     pub tags: Option<PipelineTags>,
+
+    pub severity_text: Option<String>,
+    pub severity_number: Option<u8>,
+    pub trace_id: Option<String>,
+
+    #[builder(default = default_span_id())]
+    pub span_id: Option<String>,
+}
+
+// TODO: would be removed in next pr
path: src/common/src/event/mod.rs
user: login: blaginin
body: how to do you want to do this?
created_at: 2025-04-16T10:54:52Z
author_association: NONE
diff_hunk: @@ -45,23 +45,24 @@ impl EventRecorder {
     pub fn record_event(
         &mut self,
         process_status: ProcessStatus,
-        message: String,
+        body: String,
         attributes: Option<EventAttributes>,
         timestamp: Option<DateTime<Utc>>,
     ) {
-        let event = Event {
-            timestamp: timestamp.unwrap_or_else(Utc::now),
-            message,
-            event_type: EventType::ProcessStatus,
-            process_type: ProcessType::Pipeline,
-            process_status,
-            attributes,
-            // NOTE: not a fan of constant cloning so would look for an alt
-            run_name: self.run_name.clone(),
-            run_id: self.run_id.clone(),
-            pipeline_name: self.pipeline_name.clone(),
-            tags: self.tags.clone(),
-        };
+        let event = Event::builder()
path: src/common/src/recorder.rs
user: login: blaginin
body: neat👍
created_at: 2025-04-16T10:58:02Z
author_association: NONE
diff_hunk: @@ -45,23 +45,24 @@ impl EventRecorder {
     pub fn record_event(
         &mut self,
         process_status: ProcessStatus,
-        message: String,
+        body: String,
         attributes: Option<EventAttributes>,
         timestamp: Option<DateTime<Utc>>,
     ) {
-        let event = Event {
-            timestamp: timestamp.unwrap_or_else(Utc::now),
-            message,
-            event_type: EventType::ProcessStatus,
-            process_type: ProcessType::Pipeline,
-            process_status,
-            attributes,
-            // NOTE: not a fan of constant cloning so would look for an alt
-            run_name: self.run_name.clone(),
-            run_id: self.run_id.clone(),
-            pipeline_name: self.pipeline_name.clone(),
-            tags: self.tags.clone(),
-        };
+        let event = Event::builder()
+            .body(body)
+            .timestamp(timestamp.unwrap_or_else(Utc::now))
+            .process_status(process_status)
+            .pipeline_name(self.pipeline_name.clone())
+            .run_name(self.run_name.clone())
+            .run_id(self.run_id.clone())
+            .tags(self.tags.clone())
+            .attributes(attributes)
+            .build();
+
+        self.events.push(event);
+    }
+    pub fn record_event_v2(&mut self, event: Event) {
         self.events.push(event);
     }
path: src/common/src/recorder.rs
user: login: blaginin
body: can we add this method when we actually need it? I'm not a big fan of added non-used methods - i appreciate it can be added quite quickly?
created_at: 2025-04-16T10:59:13Z
author_association: NONE
diff_hunk: @@ -163,88 +74,89 @@ impl AuroraClient {
         run_name: &str,
         run_id: &str,
         pipeline_name: &str,
-        data: impl IntoIterator<Item = &Event>,
+        logs: impl IntoIterator<Item = Event>,
path: src/client/src/exporters/db.rs
user: login: blaginin
body: not sure that would be the right long-term solution - as we do want to have many exportes but don't want to clone events for every one of them
created_at: 2025-04-16T11:04:36Z
author_association: NONE
diff_hunk: @@ -36,37 +39,181 @@ pub enum ProcessStatus {
     NextflowLogEvent,
 }
 
-impl ProcessStatus {
-    pub fn as_str(&self) -> &'static str {
+impl std::fmt::Display for ProcessStatus {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
         match self {
-            ProcessStatus::NewRun => "new_run",
-            ProcessStatus::FinishedRun => "finished_run",
-            ProcessStatus::ToolExecution => "tool_execution",
-            ProcessStatus::FinishedToolExecution => "finished_tool_execution",
-            ProcessStatus::MetricEvent => "metric_event",
-            ProcessStatus::SyslogEvent => "syslog_event",
-            ProcessStatus::ToolMetricEvent => "tool_metric_event",
-            ProcessStatus::TestEvent => "test_event", // Handle TestEvent
-            ProcessStatus::RunStatusMessage => "run_status_message",
-            ProcessStatus::Alert => "alert",
-            ProcessStatus::DataSamplesEvent => "datasets_in_process",
-            ProcessStatus::NextflowLogEvent => "nextflow_log_event",
+            ProcessStatus::NewRun => write!(f, "new_run"),
+            ProcessStatus::FinishedRun => write!(f, "finished_run"),
+            ProcessStatus::ToolExecution => write!(f, "tool_execution"),
+            ProcessStatus::FinishedToolExecution => write!(f, "finished_tool_execution"),
+            ProcessStatus::ToolMetricEvent => write!(f, "tool_metric_event"),
+            ProcessStatus::MetricEvent => write!(f, "metric_event"),
+            ProcessStatus::SyslogEvent => write!(f, "syslog_event"),
+            ProcessStatus::RunStatusMessage => write!(f, "run_status_message"),
+            ProcessStatus::Alert => write!(f, "alert"),
+            ProcessStatus::DataSamplesEvent => write!(f, "datasets_in_process"),
+            ProcessStatus::TestEvent => write!(f, "test_event"),
+            ProcessStatus::NextflowLogEvent => write!(f, "nextflow_log_event"),
         }
     }
 }
-#[derive(serde::Serialize, serde::Deserialize, Debug, Clone)]
+
+fn default_span_id() -> Option<String> {
+    Some(Uuid::new_v4().to_string())
+}
+
+#[derive(Serialize, Deserialize, Debug, Clone, TypedBuilder)]
+#[builder(field_defaults(default))]
 pub struct Event {
     #[serde(with = "ts_seconds")]
     pub timestamp: DateTime<Utc>,
-    pub message: String,
 
+    #[builder(setter(into))]
+    pub body: String,
+
+    #[builder(default = EventType::ProcessStatus)]
     pub event_type: EventType,
+
+    #[builder(default = ProcessType::Pipeline)]
     pub process_type: ProcessType,
+
+    #[builder(default = ProcessStatus::TestEvent)]
     pub process_status: ProcessStatus,
 
     pub pipeline_name: Option<String>,
     pub run_name: Option<String>,
     pub run_id: Option<String>,
     pub attributes: Option<EventAttributes>,
     pub tags: Option<PipelineTags>,
+
+    pub severity_text: Option<String>,
+    pub severity_number: Option<u8>,
+    pub trace_id: Option<String>,
+
+    #[builder(default = default_span_id())]
+    pub span_id: Option<String>,
+}
+
+// TODO: would be removed in next pr
+pub struct EventInsert {
+    pub event_timestamp: DateTime<Utc>,
+    pub body: String,
+    pub severity_text: Option<String>,
+    pub severity_number: Option<i16>,
+    pub trace_id: Option<String>,
+    pub span_id: Option<String>,
+
+    pub source_type: String,
+    pub instrumentation_version: Option<String>,
+    pub instrumentation_type: Option<String>,
+    pub environment: Option<String>,
+    pub pipeline_type: Option<String>,
+    pub user_operator: Option<String>,
+    pub organization_id: Option<String>,
+    pub department: Option<String>,
+
+    pub run_id: String,
+    pub run_name: String,
+    pub pipeline_name: String,
+    pub job_id: Option<String>,
+    pub parent_job_id: Option<String>,
+    pub child_job_ids: Option<Vec<String>>,
+    pub workflow_engine: Option<String>,
+
+    pub ec2_cost_per_hour: Option<f64>,
+    pub cpu_usage: Option<f32>,
+    pub mem_used: Option<f64>,
+    pub processed_dataset: Option<i32>,
+    pub process_status: String,
+
+    pub attributes: Value,
+    pub resource_attributes: Value,
+    pub tags: Value,
+}
+
+impl From<Event> for EventInsert {
+    fn from(event: Event) -> Self {
+        let mut attributes = json!({});
+        let mut resource_attributes = json!({});
+        let mut job_id = None;
+        let mut parent_job_id = None;
+        let mut child_job_ids = None;
+        let mut workflow_engine = None;
+        let mut cpu_usage = None;
+        let mut mem_used = None;
+        let mut ec2_cost_per_hour = None;
+        let mut processed_dataset = None;
+
+        if let Some(attr) = &event.attributes {
+            match attr {
+                EventAttributes::Process(p) => {
+                    cpu_usage = Some(p.process_cpu_utilization);
+                    mem_used = Some(p.process_memory_usage as f64);
+                    job_id = p.job_id.clone();
+                    attributes = serde_json::to_value(p).unwrap_or_default();
path: src/common/src/event/mod.rs
user: login: blaginin
body: I am not very sure about `unwrap_or_default` here -- IMO, it's better to fail loudly rather that silently swap user data with `[]`. before it used to be `TryFrom` - is there a reason you've switched to `From`?
created_at: 2025-04-16T11:07:44Z
author_association: NONE
diff_hunk: @@ -157,8 +157,8 @@ impl TracerClient {
         let run_name = self
             .current_run
             .as_ref()
-            .map(|st| st.name.as_str())
-            .unwrap_or("anonymous");
+            .map(|st| st.name.to_string())
+            .unwrap_or("anonymous".into());
path: src/client/src/tracer_client.rs
user: login: blaginin
body: what's the reason of this change? it used to be `&str` without additional allocations - but now it'd do unnecessary allocs 
created_at: 2025-04-16T11:10:07Z
author_association: NONE
diff_hunk: @@ -178,10 +178,10 @@ impl TracerClient {
 
             self.db_client
                 .batch_insert_events(
-                    run_name,
+                    &run_name,
                     run_id,
                     &self.pipeline_name,
-                    self.logs.get_events(),
+                    self.logs.get_events().iter().cloned(),
path: src/client/src/tracer_client.rs
user: login: blaginin
body: do we really want cloned here? IMO &Event is more than enough for exports
created_at: 2025-04-16T11:10:57Z
author_association: NONE
diff_hunk: @@ -36,37 +39,181 @@ pub enum ProcessStatus {
     NextflowLogEvent,
 }
 
-impl ProcessStatus {
-    pub fn as_str(&self) -> &'static str {
+impl std::fmt::Display for ProcessStatus {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
         match self {
-            ProcessStatus::NewRun => "new_run",
-            ProcessStatus::FinishedRun => "finished_run",
-            ProcessStatus::ToolExecution => "tool_execution",
-            ProcessStatus::FinishedToolExecution => "finished_tool_execution",
-            ProcessStatus::MetricEvent => "metric_event",
-            ProcessStatus::SyslogEvent => "syslog_event",
-            ProcessStatus::ToolMetricEvent => "tool_metric_event",
-            ProcessStatus::TestEvent => "test_event", // Handle TestEvent
-            ProcessStatus::RunStatusMessage => "run_status_message",
-            ProcessStatus::Alert => "alert",
-            ProcessStatus::DataSamplesEvent => "datasets_in_process",
-            ProcessStatus::NextflowLogEvent => "nextflow_log_event",
+            ProcessStatus::NewRun => write!(f, "new_run"),
+            ProcessStatus::FinishedRun => write!(f, "finished_run"),
+            ProcessStatus::ToolExecution => write!(f, "tool_execution"),
+            ProcessStatus::FinishedToolExecution => write!(f, "finished_tool_execution"),
+            ProcessStatus::ToolMetricEvent => write!(f, "tool_metric_event"),
+            ProcessStatus::MetricEvent => write!(f, "metric_event"),
+            ProcessStatus::SyslogEvent => write!(f, "syslog_event"),
+            ProcessStatus::RunStatusMessage => write!(f, "run_status_message"),
+            ProcessStatus::Alert => write!(f, "alert"),
+            ProcessStatus::DataSamplesEvent => write!(f, "datasets_in_process"),
+            ProcessStatus::TestEvent => write!(f, "test_event"),
+            ProcessStatus::NextflowLogEvent => write!(f, "nextflow_log_event"),
         }
     }
 }
-#[derive(serde::Serialize, serde::Deserialize, Debug, Clone)]
+
+fn default_span_id() -> Option<String> {
path: src/common/src/event/mod.rs
user: login: blaginin
body: 1 - default shouldn't be None I believe 😀 Let's make it return `String`
2 - may be opinionated, but you'd normally expect the _default_ value to keep the same - where's here it'll be a new value every time. Should we call it `new_span_id` instead?
created_at: 2025-04-16T11:12:35Z
author_association: NONE
diff_hunk: @@ -36,37 +39,181 @@ pub enum ProcessStatus {
     NextflowLogEvent,
 }
 
-impl ProcessStatus {
-    pub fn as_str(&self) -> &'static str {
+impl std::fmt::Display for ProcessStatus {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
         match self {
-            ProcessStatus::NewRun => "new_run",
-            ProcessStatus::FinishedRun => "finished_run",
-            ProcessStatus::ToolExecution => "tool_execution",
-            ProcessStatus::FinishedToolExecution => "finished_tool_execution",
-            ProcessStatus::MetricEvent => "metric_event",
-            ProcessStatus::SyslogEvent => "syslog_event",
-            ProcessStatus::ToolMetricEvent => "tool_metric_event",
-            ProcessStatus::TestEvent => "test_event", // Handle TestEvent
-            ProcessStatus::RunStatusMessage => "run_status_message",
-            ProcessStatus::Alert => "alert",
-            ProcessStatus::DataSamplesEvent => "datasets_in_process",
-            ProcessStatus::NextflowLogEvent => "nextflow_log_event",
+            ProcessStatus::NewRun => write!(f, "new_run"),
+            ProcessStatus::FinishedRun => write!(f, "finished_run"),
+            ProcessStatus::ToolExecution => write!(f, "tool_execution"),
+            ProcessStatus::FinishedToolExecution => write!(f, "finished_tool_execution"),
+            ProcessStatus::ToolMetricEvent => write!(f, "tool_metric_event"),
+            ProcessStatus::MetricEvent => write!(f, "metric_event"),
+            ProcessStatus::SyslogEvent => write!(f, "syslog_event"),
+            ProcessStatus::RunStatusMessage => write!(f, "run_status_message"),
+            ProcessStatus::Alert => write!(f, "alert"),
+            ProcessStatus::DataSamplesEvent => write!(f, "datasets_in_process"),
+            ProcessStatus::TestEvent => write!(f, "test_event"),
+            ProcessStatus::NextflowLogEvent => write!(f, "nextflow_log_event"),
         }
     }
 }
-#[derive(serde::Serialize, serde::Deserialize, Debug, Clone)]
+
+fn default_span_id() -> Option<String> {
+    Some(Uuid::new_v4().to_string())
+}
+
+#[derive(Serialize, Deserialize, Debug, Clone, TypedBuilder)]
+#[builder(field_defaults(default))]
 pub struct Event {
     #[serde(with = "ts_seconds")]
     pub timestamp: DateTime<Utc>,
-    pub message: String,
 
+    #[builder(setter(into))]
+    pub body: String,
path: src/common/src/event/mod.rs
user: login: blaginin
body: opinion: message was actually better, because event body is opaque. in a way, `process_status` is also event body
created_at: 2025-04-16T11:17:58Z
author_association: NONE
diff_hunk: @@ -36,37 +39,181 @@ pub enum ProcessStatus {
     NextflowLogEvent,
 }
 
-impl ProcessStatus {
-    pub fn as_str(&self) -> &'static str {
+impl std::fmt::Display for ProcessStatus {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
         match self {
-            ProcessStatus::NewRun => "new_run",
-            ProcessStatus::FinishedRun => "finished_run",
-            ProcessStatus::ToolExecution => "tool_execution",
-            ProcessStatus::FinishedToolExecution => "finished_tool_execution",
-            ProcessStatus::MetricEvent => "metric_event",
-            ProcessStatus::SyslogEvent => "syslog_event",
-            ProcessStatus::ToolMetricEvent => "tool_metric_event",
-            ProcessStatus::TestEvent => "test_event", // Handle TestEvent
-            ProcessStatus::RunStatusMessage => "run_status_message",
-            ProcessStatus::Alert => "alert",
-            ProcessStatus::DataSamplesEvent => "datasets_in_process",
-            ProcessStatus::NextflowLogEvent => "nextflow_log_event",
+            ProcessStatus::NewRun => write!(f, "new_run"),
+            ProcessStatus::FinishedRun => write!(f, "finished_run"),
+            ProcessStatus::ToolExecution => write!(f, "tool_execution"),
+            ProcessStatus::FinishedToolExecution => write!(f, "finished_tool_execution"),
+            ProcessStatus::ToolMetricEvent => write!(f, "tool_metric_event"),
+            ProcessStatus::MetricEvent => write!(f, "metric_event"),
+            ProcessStatus::SyslogEvent => write!(f, "syslog_event"),
+            ProcessStatus::RunStatusMessage => write!(f, "run_status_message"),
+            ProcessStatus::Alert => write!(f, "alert"),
+            ProcessStatus::DataSamplesEvent => write!(f, "datasets_in_process"),
+            ProcessStatus::TestEvent => write!(f, "test_event"),
+            ProcessStatus::NextflowLogEvent => write!(f, "nextflow_log_event"),
         }
     }
 }
-#[derive(serde::Serialize, serde::Deserialize, Debug, Clone)]
+
+fn default_span_id() -> Option<String> {
+    Some(Uuid::new_v4().to_string())
+}
+
+#[derive(Serialize, Deserialize, Debug, Clone, TypedBuilder)]
+#[builder(field_defaults(default))]
 pub struct Event {
     #[serde(with = "ts_seconds")]
     pub timestamp: DateTime<Utc>,
-    pub message: String,
 
+    #[builder(setter(into))]
+    pub body: String,
+
+    #[builder(default = EventType::ProcessStatus)]
     pub event_type: EventType,
+
+    #[builder(default = ProcessType::Pipeline)]
     pub process_type: ProcessType,
+
+    #[builder(default = ProcessStatus::TestEvent)]
     pub process_status: ProcessStatus,
 
     pub pipeline_name: Option<String>,
     pub run_name: Option<String>,
     pub run_id: Option<String>,
     pub attributes: Option<EventAttributes>,
     pub tags: Option<PipelineTags>,
+
+    pub severity_text: Option<String>,
+    pub severity_number: Option<u8>,
+    pub trace_id: Option<String>,
+
+    #[builder(default = default_span_id())]
+    pub span_id: Option<String>,
path: src/common/src/event/mod.rs
user: login: blaginin
body: can't it be `String`?
created_at: 2025-04-16T11:18:42Z
author_association: NONE
diff_hunk: @@ -36,37 +39,181 @@ pub enum ProcessStatus {
     NextflowLogEvent,
 }
 
-impl ProcessStatus {
-    pub fn as_str(&self) -> &'static str {
+impl std::fmt::Display for ProcessStatus {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
         match self {
-            ProcessStatus::NewRun => "new_run",
-            ProcessStatus::FinishedRun => "finished_run",
-            ProcessStatus::ToolExecution => "tool_execution",
-            ProcessStatus::FinishedToolExecution => "finished_tool_execution",
-            ProcessStatus::MetricEvent => "metric_event",
-            ProcessStatus::SyslogEvent => "syslog_event",
-            ProcessStatus::ToolMetricEvent => "tool_metric_event",
-            ProcessStatus::TestEvent => "test_event", // Handle TestEvent
-            ProcessStatus::RunStatusMessage => "run_status_message",
-            ProcessStatus::Alert => "alert",
-            ProcessStatus::DataSamplesEvent => "datasets_in_process",
-            ProcessStatus::NextflowLogEvent => "nextflow_log_event",
+            ProcessStatus::NewRun => write!(f, "new_run"),
+            ProcessStatus::FinishedRun => write!(f, "finished_run"),
+            ProcessStatus::ToolExecution => write!(f, "tool_execution"),
+            ProcessStatus::FinishedToolExecution => write!(f, "finished_tool_execution"),
+            ProcessStatus::ToolMetricEvent => write!(f, "tool_metric_event"),
+            ProcessStatus::MetricEvent => write!(f, "metric_event"),
+            ProcessStatus::SyslogEvent => write!(f, "syslog_event"),
+            ProcessStatus::RunStatusMessage => write!(f, "run_status_message"),
+            ProcessStatus::Alert => write!(f, "alert"),
+            ProcessStatus::DataSamplesEvent => write!(f, "datasets_in_process"),
+            ProcessStatus::TestEvent => write!(f, "test_event"),
+            ProcessStatus::NextflowLogEvent => write!(f, "nextflow_log_event"),
         }
     }
 }
-#[derive(serde::Serialize, serde::Deserialize, Debug, Clone)]
+
+fn default_span_id() -> Option<String> {
path: src/common/src/event/mod.rs
user: login: blaginin
body: maybe you can put is as inliner?
created_at: 2025-04-16T11:19:36Z
author_association: NONE
diff_hunk: @@ -79,7 +79,7 @@ jobs:
           TRACER_DATABASE_NAME: ${{ secrets.TRACER_DATABASE_NAME }}
           TRACER_DATABASE_HOST: ${{ secrets.TRACER_DATABASE_HOST }}
           DATABASE_URL: "postgresql://postgres:postgres@localhost:${{ job.services.postgres.ports[5432] }}/tracer_db" # localhost since VM
-        run: make test-tracer
+        run: ./scripts/migrate.sh && make test-tracer
path: .github/workflows/build-and-test.yml
user: login: MicheleVerriello
body: okay so we can just remove 
```
TRACER_DATABASE_NAME: ${{ secrets.TRACER_DATABASE_NAME }}
TRACER_DATABASE_HOST: ${{ secrets.TRACER_DATABASE_HOST }}
```
right?
created_at: 2025-04-16T11:21:45Z
author_association: CONTRIBUTOR
diff_hunk: @@ -36,37 +39,181 @@ pub enum ProcessStatus {
     NextflowLogEvent,
 }
 
-impl ProcessStatus {
-    pub fn as_str(&self) -> &'static str {
+impl std::fmt::Display for ProcessStatus {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
         match self {
-            ProcessStatus::NewRun => "new_run",
-            ProcessStatus::FinishedRun => "finished_run",
-            ProcessStatus::ToolExecution => "tool_execution",
-            ProcessStatus::FinishedToolExecution => "finished_tool_execution",
-            ProcessStatus::MetricEvent => "metric_event",
-            ProcessStatus::SyslogEvent => "syslog_event",
-            ProcessStatus::ToolMetricEvent => "tool_metric_event",
-            ProcessStatus::TestEvent => "test_event", // Handle TestEvent
-            ProcessStatus::RunStatusMessage => "run_status_message",
-            ProcessStatus::Alert => "alert",
-            ProcessStatus::DataSamplesEvent => "datasets_in_process",
-            ProcessStatus::NextflowLogEvent => "nextflow_log_event",
+            ProcessStatus::NewRun => write!(f, "new_run"),
+            ProcessStatus::FinishedRun => write!(f, "finished_run"),
+            ProcessStatus::ToolExecution => write!(f, "tool_execution"),
+            ProcessStatus::FinishedToolExecution => write!(f, "finished_tool_execution"),
+            ProcessStatus::ToolMetricEvent => write!(f, "tool_metric_event"),
+            ProcessStatus::MetricEvent => write!(f, "metric_event"),
+            ProcessStatus::SyslogEvent => write!(f, "syslog_event"),
+            ProcessStatus::RunStatusMessage => write!(f, "run_status_message"),
+            ProcessStatus::Alert => write!(f, "alert"),
+            ProcessStatus::DataSamplesEvent => write!(f, "datasets_in_process"),
+            ProcessStatus::TestEvent => write!(f, "test_event"),
+            ProcessStatus::NextflowLogEvent => write!(f, "nextflow_log_event"),
         }
     }
 }
-#[derive(serde::Serialize, serde::Deserialize, Debug, Clone)]
+
+fn default_span_id() -> Option<String> {
+    Some(Uuid::new_v4().to_string())
+}
+
+#[derive(Serialize, Deserialize, Debug, Clone, TypedBuilder)]
+#[builder(field_defaults(default))]
 pub struct Event {
     #[serde(with = "ts_seconds")]
     pub timestamp: DateTime<Utc>,
-    pub message: String,
 
+    #[builder(setter(into))]
+    pub body: String,
path: src/common/src/event/mod.rs
user: login: MicheleVerriello
body: @blaginin in regard to the `process_status`,  is quite confusing imo, need to be updated in a next task.
i'd call it `event_type` or something else similar
created_at: 2025-04-16T11:25:22Z
author_association: CONTRIBUTOR
diff_hunk: @@ -36,37 +39,181 @@ pub enum ProcessStatus {
     NextflowLogEvent,
 }
 
-impl ProcessStatus {
-    pub fn as_str(&self) -> &'static str {
+impl std::fmt::Display for ProcessStatus {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
         match self {
-            ProcessStatus::NewRun => "new_run",
-            ProcessStatus::FinishedRun => "finished_run",
-            ProcessStatus::ToolExecution => "tool_execution",
-            ProcessStatus::FinishedToolExecution => "finished_tool_execution",
-            ProcessStatus::MetricEvent => "metric_event",
-            ProcessStatus::SyslogEvent => "syslog_event",
-            ProcessStatus::ToolMetricEvent => "tool_metric_event",
-            ProcessStatus::TestEvent => "test_event", // Handle TestEvent
-            ProcessStatus::RunStatusMessage => "run_status_message",
-            ProcessStatus::Alert => "alert",
-            ProcessStatus::DataSamplesEvent => "datasets_in_process",
-            ProcessStatus::NextflowLogEvent => "nextflow_log_event",
+            ProcessStatus::NewRun => write!(f, "new_run"),
+            ProcessStatus::FinishedRun => write!(f, "finished_run"),
+            ProcessStatus::ToolExecution => write!(f, "tool_execution"),
+            ProcessStatus::FinishedToolExecution => write!(f, "finished_tool_execution"),
+            ProcessStatus::ToolMetricEvent => write!(f, "tool_metric_event"),
+            ProcessStatus::MetricEvent => write!(f, "metric_event"),
+            ProcessStatus::SyslogEvent => write!(f, "syslog_event"),
+            ProcessStatus::RunStatusMessage => write!(f, "run_status_message"),
+            ProcessStatus::Alert => write!(f, "alert"),
+            ProcessStatus::DataSamplesEvent => write!(f, "datasets_in_process"),
+            ProcessStatus::TestEvent => write!(f, "test_event"),
+            ProcessStatus::NextflowLogEvent => write!(f, "nextflow_log_event"),
         }
     }
 }
-#[derive(serde::Serialize, serde::Deserialize, Debug, Clone)]
+
+fn default_span_id() -> Option<String> {
+    Some(Uuid::new_v4().to_string())
+}
+
+#[derive(Serialize, Deserialize, Debug, Clone, TypedBuilder)]
+#[builder(field_defaults(default))]
 pub struct Event {
     #[serde(with = "ts_seconds")]
     pub timestamp: DateTime<Utc>,
-    pub message: String,
 
+    #[builder(setter(into))]
+    pub body: String,
path: src/common/src/event/mod.rs
user: login: blaginin
body: sure, process_status just an is an example of something that also fits into the definition of event body 
created_at: 2025-04-16T11:29:25Z
author_association: NONE
diff_hunk: @@ -79,7 +79,7 @@ jobs:
           TRACER_DATABASE_NAME: ${{ secrets.TRACER_DATABASE_NAME }}
           TRACER_DATABASE_HOST: ${{ secrets.TRACER_DATABASE_HOST }}
           DATABASE_URL: "postgresql://postgres:postgres@localhost:${{ job.services.postgres.ports[5432] }}/tracer_db" # localhost since VM
-        run: make test-tracer
+        run: ./scripts/migrate.sh && make test-tracer
path: .github/workflows/build-and-test.yml
user: login: blaginin
body: i believe we can't just remove them as they also need to point to the localhost db. I believe you should se them to the same value `DATABASE_URL` (just splitting into parts)
created_at: 2025-04-16T11:30:26Z
author_association: NONE
diff_hunk: @@ -79,7 +79,7 @@ jobs:
           TRACER_DATABASE_NAME: ${{ secrets.TRACER_DATABASE_NAME }}
           TRACER_DATABASE_HOST: ${{ secrets.TRACER_DATABASE_HOST }}
           DATABASE_URL: "postgresql://postgres:postgres@localhost:${{ job.services.postgres.ports[5432] }}/tracer_db" # localhost since VM
-        run: make test-tracer
+        run: ./scripts/migrate.sh && make test-tracer
path: .github/workflows/build-and-test.yml
user: login: blaginin
body: not sure https://github.com/Tracer-Cloud/tracer-client/pull/120/commits/74711f92159ef6aa8aa1220d0ed1958bea8c2003 will work, because tracer binary won't know which DB is to connect. i think the env vars should stay, just point to the localhost db.

The only thing which is unclear to me is what to do with database_arn, as local db doesn't have any arn...
created_at: 2025-04-16T11:37:43Z
author_association: NONE
diff_hunk: @@ -17,4 +17,7 @@ serde = { workspace = true }
 serde_json = { workspace = true }
 tokio = { workspace = true }
 url = { workspace = true }
-tracer_aws = { workspace = true }
\ No newline at end of file
+tracer_aws = { workspace = true }
+uuid = { workspace = true }
+
+typed-builder = "0.18"
path: src/common/Cargo.toml
user: login: Iamdavidonuh
body: i mean we could do that, but so far it's a dependency only used by common 
created_at: 2025-04-16T11:49:53Z
author_association: CONTRIBUTOR
diff_hunk: @@ -17,4 +17,7 @@ serde = { workspace = true }
 serde_json = { workspace = true }
 tokio = { workspace = true }
 url = { workspace = true }
-tracer_aws = { workspace = true }
\ No newline at end of file
+tracer_aws = { workspace = true }
+uuid = { workspace = true }
+
+typed-builder = "0.18"
path: src/common/Cargo.toml
user: login: blaginin
body: it'll be very hard to trace back when we'll need to add it somewhere else, forget it's already added in "common" and as a result finish with two versions. I think we should just always add stuff to the main toml 
created_at: 2025-04-16T11:51:48Z
author_association: NONE
diff_hunk: @@ -36,37 +39,181 @@ pub enum ProcessStatus {
     NextflowLogEvent,
 }
 
-impl ProcessStatus {
-    pub fn as_str(&self) -> &'static str {
+impl std::fmt::Display for ProcessStatus {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
         match self {
-            ProcessStatus::NewRun => "new_run",
-            ProcessStatus::FinishedRun => "finished_run",
-            ProcessStatus::ToolExecution => "tool_execution",
-            ProcessStatus::FinishedToolExecution => "finished_tool_execution",
-            ProcessStatus::MetricEvent => "metric_event",
-            ProcessStatus::SyslogEvent => "syslog_event",
-            ProcessStatus::ToolMetricEvent => "tool_metric_event",
-            ProcessStatus::TestEvent => "test_event", // Handle TestEvent
-            ProcessStatus::RunStatusMessage => "run_status_message",
-            ProcessStatus::Alert => "alert",
-            ProcessStatus::DataSamplesEvent => "datasets_in_process",
-            ProcessStatus::NextflowLogEvent => "nextflow_log_event",
+            ProcessStatus::NewRun => write!(f, "new_run"),
+            ProcessStatus::FinishedRun => write!(f, "finished_run"),
+            ProcessStatus::ToolExecution => write!(f, "tool_execution"),
+            ProcessStatus::FinishedToolExecution => write!(f, "finished_tool_execution"),
+            ProcessStatus::ToolMetricEvent => write!(f, "tool_metric_event"),
+            ProcessStatus::MetricEvent => write!(f, "metric_event"),
+            ProcessStatus::SyslogEvent => write!(f, "syslog_event"),
+            ProcessStatus::RunStatusMessage => write!(f, "run_status_message"),
+            ProcessStatus::Alert => write!(f, "alert"),
+            ProcessStatus::DataSamplesEvent => write!(f, "datasets_in_process"),
+            ProcessStatus::TestEvent => write!(f, "test_event"),
+            ProcessStatus::NextflowLogEvent => write!(f, "nextflow_log_event"),
         }
     }
 }
-#[derive(serde::Serialize, serde::Deserialize, Debug, Clone)]
+
+fn default_span_id() -> Option<String> {
+    Some(Uuid::new_v4().to_string())
+}
+
+#[derive(Serialize, Deserialize, Debug, Clone, TypedBuilder)]
+#[builder(field_defaults(default))]
 pub struct Event {
     #[serde(with = "ts_seconds")]
     pub timestamp: DateTime<Utc>,
-    pub message: String,
 
+    #[builder(setter(into))]
+    pub body: String,
+
+    #[builder(default = EventType::ProcessStatus)]
     pub event_type: EventType,
+
+    #[builder(default = ProcessType::Pipeline)]
     pub process_type: ProcessType,
+
+    #[builder(default = ProcessStatus::TestEvent)]
     pub process_status: ProcessStatus,
 
     pub pipeline_name: Option<String>,
     pub run_name: Option<String>,
     pub run_id: Option<String>,
     pub attributes: Option<EventAttributes>,
     pub tags: Option<PipelineTags>,
+
+    pub severity_text: Option<String>,
+    pub severity_number: Option<u8>,
+    pub trace_id: Option<String>,
+
+    #[builder(default = default_span_id())]
+    pub span_id: Option<String>,
+}
+
+// TODO: would be removed in next pr
path: src/common/src/event/mod.rs
user: login: Iamdavidonuh
body: i'm gonna move it to a seperate location...
Need to do some restructuring. If i added it in this PR, it would be just be noise covering the implementation
created_at: 2025-04-16T11:52:07Z
author_association: CONTRIBUTOR
diff_hunk: @@ -45,23 +45,24 @@ impl EventRecorder {
     pub fn record_event(
         &mut self,
         process_status: ProcessStatus,
-        message: String,
+        body: String,
         attributes: Option<EventAttributes>,
         timestamp: Option<DateTime<Utc>>,
     ) {
-        let event = Event {
-            timestamp: timestamp.unwrap_or_else(Utc::now),
-            message,
-            event_type: EventType::ProcessStatus,
-            process_type: ProcessType::Pipeline,
-            process_status,
-            attributes,
-            // NOTE: not a fan of constant cloning so would look for an alt
-            run_name: self.run_name.clone(),
-            run_id: self.run_id.clone(),
-            pipeline_name: self.pipeline_name.clone(),
-            tags: self.tags.clone(),
-        };
+        let event = Event::builder()
+            .body(body)
+            .timestamp(timestamp.unwrap_or_else(Utc::now))
+            .process_status(process_status)
+            .pipeline_name(self.pipeline_name.clone())
+            .run_name(self.run_name.clone())
+            .run_id(self.run_id.clone())
+            .tags(self.tags.clone())
+            .attributes(attributes)
+            .build();
+
+        self.events.push(event);
+    }
+    pub fn record_event_v2(&mut self, event: Event) {
         self.events.push(event);
     }
path: src/common/src/recorder.rs
user: login: Iamdavidonuh
body: it's gonna be used for what @MicheleVerriello is about to do
created_at: 2025-04-16T11:52:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -36,37 +39,181 @@ pub enum ProcessStatus {
     NextflowLogEvent,
 }
 
-impl ProcessStatus {
-    pub fn as_str(&self) -> &'static str {
+impl std::fmt::Display for ProcessStatus {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
         match self {
-            ProcessStatus::NewRun => "new_run",
-            ProcessStatus::FinishedRun => "finished_run",
-            ProcessStatus::ToolExecution => "tool_execution",
-            ProcessStatus::FinishedToolExecution => "finished_tool_execution",
-            ProcessStatus::MetricEvent => "metric_event",
-            ProcessStatus::SyslogEvent => "syslog_event",
-            ProcessStatus::ToolMetricEvent => "tool_metric_event",
-            ProcessStatus::TestEvent => "test_event", // Handle TestEvent
-            ProcessStatus::RunStatusMessage => "run_status_message",
-            ProcessStatus::Alert => "alert",
-            ProcessStatus::DataSamplesEvent => "datasets_in_process",
-            ProcessStatus::NextflowLogEvent => "nextflow_log_event",
+            ProcessStatus::NewRun => write!(f, "new_run"),
+            ProcessStatus::FinishedRun => write!(f, "finished_run"),
+            ProcessStatus::ToolExecution => write!(f, "tool_execution"),
+            ProcessStatus::FinishedToolExecution => write!(f, "finished_tool_execution"),
+            ProcessStatus::ToolMetricEvent => write!(f, "tool_metric_event"),
+            ProcessStatus::MetricEvent => write!(f, "metric_event"),
+            ProcessStatus::SyslogEvent => write!(f, "syslog_event"),
+            ProcessStatus::RunStatusMessage => write!(f, "run_status_message"),
+            ProcessStatus::Alert => write!(f, "alert"),
+            ProcessStatus::DataSamplesEvent => write!(f, "datasets_in_process"),
+            ProcessStatus::TestEvent => write!(f, "test_event"),
+            ProcessStatus::NextflowLogEvent => write!(f, "nextflow_log_event"),
         }
     }
 }
-#[derive(serde::Serialize, serde::Deserialize, Debug, Clone)]
+
+fn default_span_id() -> Option<String> {
+    Some(Uuid::new_v4().to_string())
+}
+
+#[derive(Serialize, Deserialize, Debug, Clone, TypedBuilder)]
+#[builder(field_defaults(default))]
 pub struct Event {
     #[serde(with = "ts_seconds")]
     pub timestamp: DateTime<Utc>,
-    pub message: String,
 
+    #[builder(setter(into))]
+    pub body: String,
+
+    #[builder(default = EventType::ProcessStatus)]
     pub event_type: EventType,
+
+    #[builder(default = ProcessType::Pipeline)]
     pub process_type: ProcessType,
+
+    #[builder(default = ProcessStatus::TestEvent)]
     pub process_status: ProcessStatus,
 
     pub pipeline_name: Option<String>,
     pub run_name: Option<String>,
     pub run_id: Option<String>,
     pub attributes: Option<EventAttributes>,
     pub tags: Option<PipelineTags>,
+
+    pub severity_text: Option<String>,
+    pub severity_number: Option<u8>,
+    pub trace_id: Option<String>,
+
+    #[builder(default = default_span_id())]
+    pub span_id: Option<String>,
+}
+
+// TODO: would be removed in next pr
path: src/common/src/event/mod.rs
user: login: blaginin
body: you going to move or remove it?
created_at: 2025-04-16T11:53:06Z
author_association: NONE
diff_hunk: @@ -45,23 +45,24 @@ impl EventRecorder {
     pub fn record_event(
         &mut self,
         process_status: ProcessStatus,
-        message: String,
+        body: String,
         attributes: Option<EventAttributes>,
         timestamp: Option<DateTime<Utc>>,
     ) {
-        let event = Event {
-            timestamp: timestamp.unwrap_or_else(Utc::now),
-            message,
-            event_type: EventType::ProcessStatus,
-            process_type: ProcessType::Pipeline,
-            process_status,
-            attributes,
-            // NOTE: not a fan of constant cloning so would look for an alt
-            run_name: self.run_name.clone(),
-            run_id: self.run_id.clone(),
-            pipeline_name: self.pipeline_name.clone(),
-            tags: self.tags.clone(),
-        };
+        let event = Event::builder()
+            .body(body)
+            .timestamp(timestamp.unwrap_or_else(Utc::now))
+            .process_status(process_status)
+            .pipeline_name(self.pipeline_name.clone())
+            .run_name(self.run_name.clone())
+            .run_id(self.run_id.clone())
+            .tags(self.tags.clone())
+            .attributes(attributes)
+            .build();
+
+        self.events.push(event);
+    }
+    pub fn record_event_v2(&mut self, event: Event) {
         self.events.push(event);
     }
path: src/common/src/recorder.rs
user: login: blaginin
body: can Michele add it then? 
created_at: 2025-04-16T11:59:05Z
author_association: NONE
diff_hunk: @@ -45,23 +45,24 @@ impl EventRecorder {
     pub fn record_event(
         &mut self,
         process_status: ProcessStatus,
-        message: String,
+        body: String,
         attributes: Option<EventAttributes>,
         timestamp: Option<DateTime<Utc>>,
     ) {
-        let event = Event {
-            timestamp: timestamp.unwrap_or_else(Utc::now),
-            message,
-            event_type: EventType::ProcessStatus,
-            process_type: ProcessType::Pipeline,
-            process_status,
-            attributes,
-            // NOTE: not a fan of constant cloning so would look for an alt
-            run_name: self.run_name.clone(),
-            run_id: self.run_id.clone(),
-            pipeline_name: self.pipeline_name.clone(),
-            tags: self.tags.clone(),
-        };
+        let event = Event::builder()
+            .body(body)
+            .timestamp(timestamp.unwrap_or_else(Utc::now))
+            .process_status(process_status)
+            .pipeline_name(self.pipeline_name.clone())
+            .run_name(self.run_name.clone())
+            .run_id(self.run_id.clone())
+            .tags(self.tags.clone())
+            .attributes(attributes)
+            .build();
+
+        self.events.push(event);
+    }
+    pub fn record_event_v2(&mut self, event: Event) {
         self.events.push(event);
     }
path: src/common/src/recorder.rs
user: login: blaginin
body: i want to ensure unused warnings in clippy 
created_at: 2025-04-16T12:15:46Z
author_association: NONE
diff_hunk: @@ -79,7 +79,7 @@ jobs:
           TRACER_DATABASE_NAME: ${{ secrets.TRACER_DATABASE_NAME }}
           TRACER_DATABASE_HOST: ${{ secrets.TRACER_DATABASE_HOST }}
           DATABASE_URL: "postgresql://postgres:postgres@localhost:${{ job.services.postgres.ports[5432] }}/tracer_db" # localhost since VM
-        run: make test-tracer
+        run: ./scripts/migrate.sh && make test-tracer
path: .github/workflows/build-and-test.yml
user: login: MicheleVerriello
body: yes, doing some tests

created_at: 2025-04-16T12:20:07Z
author_association: CONTRIBUTOR
diff_hunk: @@ -79,7 +79,7 @@ jobs:
           TRACER_DATABASE_NAME: ${{ secrets.TRACER_DATABASE_NAME }}
           TRACER_DATABASE_HOST: ${{ secrets.TRACER_DATABASE_HOST }}
           DATABASE_URL: "postgresql://postgres:postgres@localhost:${{ job.services.postgres.ports[5432] }}/tracer_db" # localhost since VM
-        run: make test-tracer
+        run: ./scripts/migrate.sh && make test-tracer
path: .github/workflows/build-and-test.yml
user: login: Iamdavidonuh
body: cc @blaginin 
created_at: 2025-04-16T12:22:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -79,7 +79,7 @@ jobs:
           TRACER_DATABASE_NAME: ${{ secrets.TRACER_DATABASE_NAME }}
           TRACER_DATABASE_HOST: ${{ secrets.TRACER_DATABASE_HOST }}
           DATABASE_URL: "postgresql://postgres:postgres@localhost:${{ job.services.postgres.ports[5432] }}/tracer_db" # localhost since VM
-        run: make test-tracer
+        run: ./scripts/migrate.sh && make test-tracer
path: .github/workflows/build-and-test.yml
user: login: blaginin
body: lmk if you need help, otherwise will re-review when you say it's ready
created_at: 2025-04-16T12:23:09Z
author_association: NONE
diff_hunk: @@ -36,37 +39,181 @@ pub enum ProcessStatus {
     NextflowLogEvent,
 }
 
-impl ProcessStatus {
-    pub fn as_str(&self) -> &'static str {
+impl std::fmt::Display for ProcessStatus {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
         match self {
-            ProcessStatus::NewRun => "new_run",
-            ProcessStatus::FinishedRun => "finished_run",
-            ProcessStatus::ToolExecution => "tool_execution",
-            ProcessStatus::FinishedToolExecution => "finished_tool_execution",
-            ProcessStatus::MetricEvent => "metric_event",
-            ProcessStatus::SyslogEvent => "syslog_event",
-            ProcessStatus::ToolMetricEvent => "tool_metric_event",
-            ProcessStatus::TestEvent => "test_event", // Handle TestEvent
-            ProcessStatus::RunStatusMessage => "run_status_message",
-            ProcessStatus::Alert => "alert",
-            ProcessStatus::DataSamplesEvent => "datasets_in_process",
-            ProcessStatus::NextflowLogEvent => "nextflow_log_event",
+            ProcessStatus::NewRun => write!(f, "new_run"),
+            ProcessStatus::FinishedRun => write!(f, "finished_run"),
+            ProcessStatus::ToolExecution => write!(f, "tool_execution"),
+            ProcessStatus::FinishedToolExecution => write!(f, "finished_tool_execution"),
+            ProcessStatus::ToolMetricEvent => write!(f, "tool_metric_event"),
+            ProcessStatus::MetricEvent => write!(f, "metric_event"),
+            ProcessStatus::SyslogEvent => write!(f, "syslog_event"),
+            ProcessStatus::RunStatusMessage => write!(f, "run_status_message"),
+            ProcessStatus::Alert => write!(f, "alert"),
+            ProcessStatus::DataSamplesEvent => write!(f, "datasets_in_process"),
+            ProcessStatus::TestEvent => write!(f, "test_event"),
+            ProcessStatus::NextflowLogEvent => write!(f, "nextflow_log_event"),
         }
     }
 }
-#[derive(serde::Serialize, serde::Deserialize, Debug, Clone)]
+
+fn default_span_id() -> Option<String> {
+    Some(Uuid::new_v4().to_string())
+}
+
+#[derive(Serialize, Deserialize, Debug, Clone, TypedBuilder)]
+#[builder(field_defaults(default))]
 pub struct Event {
     #[serde(with = "ts_seconds")]
     pub timestamp: DateTime<Utc>,
-    pub message: String,
 
+    #[builder(setter(into))]
+    pub body: String,
+
+    #[builder(default = EventType::ProcessStatus)]
     pub event_type: EventType,
+
+    #[builder(default = ProcessType::Pipeline)]
     pub process_type: ProcessType,
+
+    #[builder(default = ProcessStatus::TestEvent)]
     pub process_status: ProcessStatus,
 
     pub pipeline_name: Option<String>,
     pub run_name: Option<String>,
     pub run_id: Option<String>,
     pub attributes: Option<EventAttributes>,
     pub tags: Option<PipelineTags>,
+
+    pub severity_text: Option<String>,
+    pub severity_number: Option<u8>,
+    pub trace_id: Option<String>,
+
+    #[builder(default = default_span_id())]
+    pub span_id: Option<String>,
+}
+
+// TODO: would be removed in next pr
+pub struct EventInsert {
+    pub event_timestamp: DateTime<Utc>,
+    pub body: String,
+    pub severity_text: Option<String>,
+    pub severity_number: Option<i16>,
+    pub trace_id: Option<String>,
+    pub span_id: Option<String>,
+
+    pub source_type: String,
+    pub instrumentation_version: Option<String>,
+    pub instrumentation_type: Option<String>,
+    pub environment: Option<String>,
+    pub pipeline_type: Option<String>,
+    pub user_operator: Option<String>,
+    pub organization_id: Option<String>,
+    pub department: Option<String>,
+
+    pub run_id: String,
+    pub run_name: String,
+    pub pipeline_name: String,
+    pub job_id: Option<String>,
+    pub parent_job_id: Option<String>,
+    pub child_job_ids: Option<Vec<String>>,
+    pub workflow_engine: Option<String>,
+
+    pub ec2_cost_per_hour: Option<f64>,
+    pub cpu_usage: Option<f32>,
+    pub mem_used: Option<f64>,
+    pub processed_dataset: Option<i32>,
+    pub process_status: String,
+
+    pub attributes: Value,
+    pub resource_attributes: Value,
+    pub tags: Value,
+}
+
+impl From<Event> for EventInsert {
+    fn from(event: Event) -> Self {
+        let mut attributes = json!({});
+        let mut resource_attributes = json!({});
+        let mut job_id = None;
+        let mut parent_job_id = None;
+        let mut child_job_ids = None;
+        let mut workflow_engine = None;
+        let mut cpu_usage = None;
+        let mut mem_used = None;
+        let mut ec2_cost_per_hour = None;
+        let mut processed_dataset = None;
+
+        if let Some(attr) = &event.attributes {
+            match attr {
+                EventAttributes::Process(p) => {
+                    cpu_usage = Some(p.process_cpu_utilization);
+                    mem_used = Some(p.process_memory_usage as f64);
+                    job_id = p.job_id.clone();
+                    attributes = serde_json::to_value(p).unwrap_or_default();
path: src/common/src/event/mod.rs
user: login: Iamdavidonuh
body: Did you implement tryfrom or a custom try_new. But yeah moved to a try from?
created_at: 2025-04-16T13:15:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -163,88 +74,89 @@ impl AuroraClient {
         run_name: &str,
         run_id: &str,
         pipeline_name: &str,
-        data: impl IntoIterator<Item = &Event>,
+        logs: impl IntoIterator<Item = Event>,
     ) -> Result<()> {
-        let now = Instant::now();
+        let now = std::time::Instant::now();
 
         const QUERY: &str = "INSERT INTO batch_jobs_logs (
-            data, job_id, run_name, run_id, pipeline_name, nextflow_session_uuid, job_ids,
-            tags, event_timestamp, ec2_cost_per_hour, cpu_usage, mem_used, processed_dataset,
-            process_status
-            )"; // when updating query, also update params
-        const PARAMS: usize = 13;
+                timestamp, body, severity_text, severity_number,
+                trace_id, span_id,
+                source_type, instrumentation_version, instrumentation_type,
+                environment, pipeline_type, user_operator, organization_id, department,
+                run_id, run_name, pipeline_name,
+                job_id, parent_job_id, child_job_ids, workflow_engine,
+                ec2_cost_per_hour, cpu_usage, mem_used, processed_dataset,
+                process_status,
+                attributes, resource_attributes, tags
+            )";
+        // when updating query, also update params
+        const PARAMS: usize = 29;
 
         fn _push_tuple(mut b: Separated<Postgres, &str>, event: EventInsert) {
-            b.push_bind(event.json_data)
-                .push_bind(event.job_id)
-                .push_bind(event.run_name)
+            b.push_bind(event.event_timestamp.naive_utc())
path: src/client/src/exporters/db.rs
user: login: blaginin
body: i couldn't find one... :(
created_at: 2025-04-16T13:29:58Z
author_association: NONE
diff_hunk: @@ -20,7 +20,7 @@ async fn test_queries_works() {
     let log_handle = tokio::spawn({
         let docker = docker.clone();
         async move {
-            common::print_all_container_logs(&docker).await;
+            // common::print_all_container_logs(&docker).await;
path: integration_tests/tests/simple_queries.rs
user: login: blaginin
body: i think it's a useful step for debugging -- lets keep it?
created_at: 2025-04-17T11:03:44Z
author_association: NONE
diff_hunk: @@ -20,7 +20,7 @@ async fn test_queries_works() {
     let log_handle = tokio::spawn({
         let docker = docker.clone();
         async move {
-            common::print_all_container_logs(&docker).await;
+            // common::print_all_container_logs(&docker).await;
path: integration_tests/tests/simple_queries.rs
user: login: Iamdavidonuh
body: @blaginin  Hey! Thanks for jumping in. Just a quick heads-up — the section you mentioned was commented out temporarily while I debug a CI/CD issue. I wrote that part and fully understand its purpose — it was just generating a bit too much stdout noise during test runs.

I do plan to add it back (as noted in the commit message), so feel free to hold off on suggestions around it for now until the debugging’s wrapped up. Appreciate the quick feedback though!
created_at: 2025-04-17T11:41:38Z
author_association: CONTRIBUTOR
diff_hunk: @@ -9,6 +9,7 @@ authors = { workspace = true }
 [dependencies]
 sqlx = { workspace = true }
 tokio = { workspace = true }
+futures-util = "0.3"
path: integration_tests/Cargo.toml
user: login: blaginin
body: can we use workspace verion please
created_at: 2025-04-17T13:22:43Z
author_association: NONE
diff_hunk: @@ -72,11 +72,11 @@ jobs:
           AWS_ENDPOINT: "https://s3.us-east-2.amazonaws.com"
           RUST_LOG: "debug"
           RUST_BACKTRACE: 1
-          TRACER_DATABASE_SECRETS_ARN: ${{ secrets.TRACER_DATABASE_SECRETS_ARN }}
+          TRACER_DATABASE_SECRETS_ARN: ${{ secrets.TRACER_DATABASE_LOCAL_SECRETS_ARN }}
path: .github/workflows/build-and-test.yml
user: login: blaginin
body: i don't think you should use env for it - just hardcode them here? 
created_at: 2025-04-17T14:25:06Z
author_association: NONE
diff_hunk: @@ -72,11 +72,11 @@ jobs:
           AWS_ENDPOINT: "https://s3.us-east-2.amazonaws.com"
           RUST_LOG: "debug"
           RUST_BACKTRACE: 1
-          TRACER_DATABASE_SECRETS_ARN: ${{ secrets.TRACER_DATABASE_SECRETS_ARN }}
+          TRACER_DATABASE_SECRETS_ARN: ${{ secrets.TRACER_DATABASE_LOCAL_SECRETS_ARN }}
path: .github/workflows/build-and-test.yml
user: login: blaginin
body: if you decide not to do that, you also need to update dependabot secrets as those are stored separately
created_at: 2025-04-17T14:25:32Z
author_association: NONE
diff_hunk: @@ -72,11 +72,11 @@ jobs:
           AWS_ENDPOINT: "https://s3.us-east-2.amazonaws.com"
           RUST_LOG: "debug"
           RUST_BACKTRACE: 1
-          TRACER_DATABASE_SECRETS_ARN: ${{ secrets.TRACER_DATABASE_SECRETS_ARN }}
+          TRACER_DATABASE_SECRETS_ARN: ${{ secrets.TRACER_DATABASE_LOCAL_SECRETS_ARN }}
path: .github/workflows/build-and-test.yml
user: login: Iamdavidonuh
body: cc: @MicheleVerriello . He made those updates
created_at: 2025-04-17T14:41:22Z
author_association: CONTRIBUTOR
diff_hunk: @@ -72,11 +72,11 @@ jobs:
           AWS_ENDPOINT: "https://s3.us-east-2.amazonaws.com"
           RUST_LOG: "debug"
           RUST_BACKTRACE: 1
-          TRACER_DATABASE_SECRETS_ARN: ${{ secrets.TRACER_DATABASE_SECRETS_ARN }}
+          TRACER_DATABASE_SECRETS_ARN: ${{ secrets.TRACER_DATABASE_LOCAL_SECRETS_ARN }}
path: .github/workflows/build-and-test.yml
user: login: MicheleVerriello
body: yes after all changes i'll update it
created_at: 2025-04-17T14:58:05Z
author_association: CONTRIBUTOR

PR comments:
user: login: cloudflare-workers-and-pages[bot]
created_at: 2025-04-15T00:42:25Z
body: ## Deploying tracer-client with &nbsp;<a href="https://pages.dev"><img alt="Cloudflare Pages" src="https://user-images.githubusercontent.com/23264/106598434-9e719e00-654f-11eb-9e59-6167043cfa01.png" width="16"></a> &nbsp;Cloudflare Pages

<table><tr><td><strong>Latest commit:</strong> </td><td>
<code>c03ad61</code>
</td></tr>
<tr><td><strong>Status:</strong></td><td>&nbsp;✅&nbsp; Deploy successful!</td></tr>
<tr><td><strong>Preview URL:</strong></td><td>
<a href='https://88681fb4.tracer-client.pages.dev'>https://88681fb4.tracer-client.pages.dev</a>
</td></tr>
<tr><td><strong>Branch Preview URL:</strong></td><td>
<a href='https://eng-165-otel-db-structure.tracer-client.pages.dev'>https://eng-165-otel-db-structure.tracer-client.pages.dev</a>
</td></tr>
</table>

[View logs](https://dash.cloudflare.com/6515221c5a196753a377299d3888771d/pages/view/tracer-client/88681fb4-857b-44fc-ab94-ab6b9f39816e)

author_association: NONE
user: login: MicheleVerriello
created_at: 2025-04-17T11:34:58Z
body: @Iamdavidonuh will create a new linear task on integration tests failing inside the actions. the tests works in local on both my and his machines
author_association: CONTRIBUTOR
user: login: blaginin
created_at: 2025-04-17T11:40:16Z
body: Let's keep the CI always green on main. Do you need help fixing integration tests before the merge? 
author_association: NONE
user: login: Iamdavidonuh
created_at: 2025-04-17T13:08:17Z
body: The root cause was traced to the missing environment variables for AWS credentials in the .env file used during CI. 
author_association: CONTRIBUTOR
user: login: blaginin
created_at: 2025-04-17T14:27:07Z
body: Also, it worries me a bit that for such a big file there are almost no test changes... 
author_association: NONE
user: login: Iamdavidonuh
created_at: 2025-04-17T14:40:46Z
body: > Also, it worries me a bit that for such a big file there are almost no test changes...

@blaginin what test cases did you think of adding that the existing tests didn't solve. The core functionality was schema changes. Tests would be around insertion and querying. Which was updated to match the schema changes. OR am i missing something else?
author_association: CONTRIBUTOR
user: login: blaginin
created_at: 2025-04-17T16:17:00Z
body: So the issue is that we're currently not tesing that the data is inserted correctly. I think if we had https://github.com/Tracer-Cloud/tracer-client/issues/105, It'd bee quite easy to update - you'd need to assert that the new columns in the db are correct
author_association: NONE