Repository: boom-astro/boom
Pull request #97: Filter Worker - output to Kafka + LSST bugfixes
Author: login: Theodlz
Created: 2025-03-31T23:11:57Z | Merged: 2025-04-14T18:34:16Z

In this PR that addresses #93, we (for both ZTF & LSST):
- combine filter results per candid (meaning when an alert passes N filters, we combine the output of all the filters for that candid in a vec)
- build alert packet for those passing at least one filter, containing metadata, photometry, cutouts, and filter results
- serialize to avro and send to kafka
- add some very basic unit tests for running filters on both surveys
- minor: also avoid some unwraps and add additional error handling
- minor: rename `docker-compose.yaml` to `docker-compose.default.yaml` so the user can just make a copy (without the default) and make edits without git telling you a file has been modified, which gets pretty annoying. The readme was edited to mention that a `cp` is needed now.

With this PR, I'm easily able to make use of BOOM to identify transients in the ZTF alert stream, and ingest the output in a marshal (SkyPortal). In a way (though we still need a lot of work in the ML side of things, and crossmatching between surveys), once that PR is tested for LSST (I only tried it for ZTF so far) and merged we'll have a fully functioning alert broker! I can already use it to look for fast transients in the ZTF alert stream (using the demo, and somewhat basic filter, recently added in `data/filters/fast.json`) which is a big win!


PS: I'm opening an issue because in a lot of places (and that includes this PR) we have host + port hardcoded for redis and kafka when it should be reading from the config, and the `conf` module should contain functions that do this + return the connections (where applicable).


Diff:
diff --git a/.github/workflows/test.yaml b/.github/workflows/test.yaml
index 94b56bfa..1722f6ec 100644
--- a/.github/workflows/test.yaml
+++ b/.github/workflows/test.yaml
@@ -29,6 +29,8 @@ jobs:
     - name: Configure cache
       uses: Swatinem/rust-cache@v2
     - name: Start services
-      run: docker compose up --build -d
+      run: |
+        cp docker-compose.default.yaml docker-compose.yaml
+        docker compose up --build -d
     - name: Tests
       run: cargo test --release
diff --git a/.gitignore b/.gitignore
index 7321c18c..c4747aec 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,10 +1,9 @@
 /target
 Cargo.lock
 config.yaml
-data/alerts/ztf/20**/*.avro
-data/alerts/ztf/20*.avro
-data/alerts/ztf/20*/*.tar.gz
-data/alerts/ztf/20*.tar.gz
+docker-compose.yaml
+data/alerts/ztf
+data/alerts/lsst
 __pycache__
 /env
 .DS_Store
diff --git a/README.md b/README.md
index 8ad33c95..ae3b1e2d 100644
--- a/README.md
+++ b/README.md
@@ -49,13 +49,17 @@ BOOM runs on macOS and Linux. You'll need:
     ```bash
     cp config.default.yaml config.yaml
     ```
-3. Launch `Valkey`, `MongoDB`, and `Kafka` using docker, using the provided `docker compose.yaml` file:
+3. Same for the `docker-compose.yaml` file:
+    ```bash
+    cp docker-compose.default.yaml docker-compose.yaml
+    ```
+4. Launch `Valkey`, `MongoDB`, and `Kafka` using docker, using the provided `docker compose.yaml` file:
     ```bash
     docker compose up -d
     ```
     This may take a couple of minutes the first time you run it, as it needs to download the docker image for each service.
     *To check if the containers are running and healthy, run `docker ps`.*
-4. Last but not least, build the Rust binaries. You can do this with or without the `--release` flag, but we recommend using it for better performance:
+5. Last but not least, build the Rust binaries. You can do this with or without the `--release` flag, but we recommend using it for better performance:
     ```bash
     cargo build --release
     ```
diff --git a/data/filters/lsst.json b/data/filters/lsst.json
new file mode 100644
index 00000000..a5397a66
--- /dev/null
+++ b/data/filters/lsst.json
@@ -0,0 +1,15 @@
+[
+  {
+    "$match": {
+      "candidate.reliability": { "$gt": 0.8 },
+      "candidate.snr": { "$gt": 5.0 },
+      "candidate.magpsf": { "$lte": 18.5 }
+    }
+  },
+  {
+    "$project": {
+      "objectId": 1,
+      "annotations.mag_now": { "$round": ["$candidate.magpsf", 2] }
+    }
+  }
+]
diff --git a/docker-compose.yaml b/docker-compose.default.yaml
similarity index 100%
rename from docker-compose.yaml
rename to docker-compose.default.yaml
diff --git a/src/filter/base.rs b/src/filter/base.rs
index d5074ca7..e1ed4dfc 100644
--- a/src/filter/base.rs
+++ b/src/filter/base.rs
@@ -1,10 +1,67 @@
+use apache_avro::Schema;
+use apache_avro::{serde_avro_bytes, Writer};
 use futures::stream::StreamExt;
 use mongodb::bson::{doc, Document};
+use rdkafka::producer::FutureProducer;
+use rdkafka::{config::ClientConfig, producer::FutureRecord};
+use redis::AsyncCommands;
+use std::num::NonZero;
 use tokio::sync::mpsc;
-use tracing::error;
+use tokio::sync::mpsc::error::TryRecvError;
+use tracing::{error, info, trace, warn};
 
 use crate::utils::worker::WorkerCmd;
 
+// This is the schema of the avro object that we will send to kafka
+// that includes the alert data and filter results
+const ALERT_SCHEMA: &str = r#"
+{
+    "type": "record",
+    "name": "Alert",
+    "fields": [
+        {"name": "candid", "type": "long"},
+        {"name": "objectId", "type": "string"},
+        {"name": "jd", "type": "double"},
+        {"name": "ra", "type": "double"},
+        {"name": "dec", "type": "double"},
+        {"name": "filters", "type": {
+            "type": "array",
+            "items": {
+                "type": "record",
+                "name": "FilterResults",
+                "fields": [
+                    {"name": "filter_id", "type": "int"},
+                    {"name": "passed_at", "type": "double"},
+                    {"name": "annotations", "type": "string"}
+                ]
+            }
+        }},
+        {"name": "photometry", "type": {
+            "type": "array",
+            "items": {
+                "type": "record",
+                "name": "Photometry",
+                "fields": [
+                    {"name": "jd", "type": "double"},
+                    {"name": "flux",  "type": ["null", "double"]},
+                    {"name": "flux_err",  "type":"double"},
+                    {"name":"band","type":"string"},
+                    {"name":"zero_point","type":"double"},
+                    {"name":"origin","type": "string"},
+                    {"name":"programid","type":"int"},
+                    {"name":"survey","type":"string"},
+                    {"name":"ra","type":["null","double"]},
+                    {"name":"dec","type":["null","double"]}
+                ]
+            }
+        }},
+        {"name":"cutoutScience","type":{"type":"bytes"}},
+        {"name":"cutoutTemplate","type":{"type":"bytes"}},
+        {"name":"cutoutDifference","type":{"type":"bytes"}}
+    ]
+}
+"#;
+
 #[derive(thiserror::Error, Debug)]
 pub enum FilterError {
     #[error("failed to retrieve filter from database")]
@@ -31,6 +88,129 @@ pub enum FilterError {
     InvalidFilterId,
 }
 
+pub fn parse_programid_candid_tuple(tuple_str: &str) -> Option<(i32, i64)> {
+    // We know that we have the programid first, followed by a comma, and then the candid.
+    // the programid is always a single digit (0-9) and the candid is a larger number.
+    // so we don't know to look for the comma to split the string.
+    // and can directly use the indexes to read the values.
+    // while this makes it very specific to this format, it is twice as fast.
+    let first_part = &tuple_str[0..1];
+    let second_part = &tuple_str[2..];
+    let first = first_part.parse::<i32>();
+    let second = second_part.parse::<i64>();
+    if let (Ok(first_value), Ok(second_value)) = (first, second) {
+        return Some((first_value, second_value));
+    }
+    None
+}
+
+#[derive(Debug, serde::Serialize, serde::Deserialize)]
+pub enum Origin {
+    Alert,
+    ForcedPhot,
+}
+
+#[derive(Debug, serde::Serialize, serde::Deserialize)]
+pub enum Survey {
+    ZTF,
+    LSST,
+}
+
+#[derive(Debug, serde::Serialize, serde::Deserialize)]
+pub struct Photometry {
+    pub jd: f64,
+    pub flux: Option<f64>,
+    pub flux_err: f64,
+    pub band: String,
+    pub zero_point: f64,
+    pub origin: Origin,
+    pub programid: i32,
+    pub survey: Survey,
+    pub ra: Option<f64>,
+    pub dec: Option<f64>,
+}
+
+#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]
+pub struct FilterResults {
+    pub filter_id: i32,
+    pub passed_at: f64, // timestamp in seconds
+    pub annotations: String,
+}
+
+#[derive(Debug, serde::Serialize, serde::Deserialize)]
+pub struct Alert {
+    pub candid: i64,
+    #[serde(rename = "objectId")]
+    pub object_id: String,
+    pub jd: f64,
+    pub ra: f64,
+    pub dec: f64,
+    pub filters: Vec<FilterResults>,
+    pub photometry: Vec<Photometry>,
+    #[serde(with = "serde_avro_bytes", rename = "cutoutScience")]
+    pub cutout_science: Vec<u8>,
+    #[serde(with = "serde_avro_bytes", rename = "cutoutTemplate")]
+    pub cutout_template: Vec<u8>,
+    #[serde(with = "serde_avro_bytes", rename = "cutoutDifference")]
+    pub cutout_difference: Vec<u8>,
+}
+
+pub fn load_alert_schema() -> Result<Schema, FilterWorkerError> {
+    let schema = Schema::parse_str(ALERT_SCHEMA).map_err(|e| {
+        error!("Failed to parse alert schema: {}", e);
+        FilterWorkerError::LoadAlertSchemaError(e)
+    })?;
+
+    Ok(schema)
+}
+
+pub fn alert_to_avro_bytes(alert: &Alert, schema: &Schema) -> Result<Vec<u8>, FilterWorkerError> {
+    let mut writer = Writer::new(schema, Vec::new());
+    writer.append_ser(alert).map_err(|e| {
+        error!("Failed to serialize alert to Avro: {}", e);
+        FilterWorkerError::SerializeAlertError(e)
+    })?;
+    let encoded = writer.into_inner().map_err(|e| {
+        error!("Failed to finalize Avro writer: {}", e);
+        FilterWorkerError::SerializeAlertError(e)
+    })?;
+
+    Ok(encoded)
+}
+
+// TODO, use the config file to get the kafka server
+pub async fn create_producer() -> Result<FutureProducer, FilterWorkerError> {
+    let producer: FutureProducer = ClientConfig::new()
+        .set("bootstrap.servers", "localhost:9092")
+        .set("message.timeout.ms", "5000")
+        .create()
+        .map_err(FilterWorkerError::KafkaProducerError)?;
+
+    Ok(producer)
+}
+
+pub async fn send_alert_to_kafka(
+    alert: &Alert,
+    schema: &Schema,
+    producer: &FutureProducer,
+    topic: &str,
+    id: &str,
+) -> Result<(), FilterWorkerError> {
+    let encoded = alert_to_avro_bytes(alert, schema)?;
+
+    let record = FutureRecord::to(&topic).key(id).payload(&encoded);
+
+    producer
+        .send(record, std::time::Duration::from_secs(0))
+        .await
+        .map_err(|(e, _)| {
+            warn!("Failed to send filter result to Kafka: {}", e);
+            FilterWorkerError::SendKafkaMessageError(e)
+        })?;
+
+    Ok(())
+}
+
 pub async fn get_filter_object(
     filter_id: i32,
     catalog: &str,
@@ -96,7 +276,7 @@ pub async fn get_filter_object(
     Ok(filter_obj)
 }
 
-pub async fn process_alerts(
+pub async fn run_filter(
     candids: Vec<i64>,
     mut pipeline: Vec<Document>,
     alert_collection: &mongodb::Collection<Document>,
@@ -165,27 +345,121 @@ pub enum FilterWorkerError {
     GetQueueNameError,
     #[error("failed to get filter by queue")]
     GetFilterByQueueError,
+    #[error("failed to send filter results to kafka")]
+    SendKafkaMessageError(#[source] rdkafka::error::KafkaError),
+    #[error("could not find alert by candid")]
+    GetAlertByCandidError(#[source] mongodb::error::Error),
+    #[error("could not find alert")]
+    AlertNotFound,
+    #[error("could not load alert schema")]
+    LoadAlertSchemaError(#[source] apache_avro::Error),
+    #[error("failed to serialize alert to Avro")]
+    SerializeAlertError(#[source] apache_avro::Error),
+    #[error("could not find document field")]
+    MissingFieldError(#[from] mongodb::bson::document::ValueAccessError),
+    #[error("failed to create kafka producer")]
+    KafkaProducerError(#[source] rdkafka::error::KafkaError),
 }
 
 #[async_trait::async_trait]
 pub trait FilterWorker {
-    async fn new(
-        id: String,
-        receiver: mpsc::Receiver<WorkerCmd>,
-        config_path: &str,
-    ) -> Result<Self, FilterWorkerError>
+    async fn new(config_path: &str) -> Result<Self, FilterWorkerError>
     where
         Self: Sized;
-    async fn run(&mut self) -> Result<(), FilterWorkerError>;
+    fn input_queue_name(&self) -> String;
+    fn output_topic_name(&self) -> String;
+    fn has_filters(&self) -> bool;
+    async fn build_alert(
+        &self,
+        candid: i64,
+        filter_results: Vec<FilterResults>,
+    ) -> Result<Alert, FilterWorkerError>;
+    async fn process_alerts(&mut self, alerts: &[String]) -> Result<Vec<Alert>, FilterWorkerError>;
 }
 
 #[tokio::main]
 pub async fn run_filter_worker<T: FilterWorker>(
     id: String,
-    receiver: mpsc::Receiver<WorkerCmd>,
+    mut receiver: mpsc::Receiver<WorkerCmd>,
     config_path: &str,
 ) -> Result<(), FilterWorkerError> {
-    let mut filter_worker = T::new(id, receiver, config_path).await?;
-    filter_worker.run().await?;
+    let mut filter_worker = T::new(config_path).await?;
+
+    if !filter_worker.has_filters() {
+        info!(
+            "No filters available for filter worker {} to process alerts",
+            &id
+        );
+        return Ok(());
+    }
+
+    // in a never ending loop, loop over the queues
+    let client_redis = redis::Client::open("redis://localhost:6379".to_string())
+        .map_err(FilterWorkerError::ConnectRedisError)?;
+    let mut con = client_redis
+        .get_multiplexed_async_connection()
+        .await
+        .map_err(FilterWorkerError::ConnectRedisError)?;
+
+    let input_queue = filter_worker.input_queue_name();
+    let output_topic = filter_worker.output_topic_name();
+
+    let producer = create_producer().await?;
+    let schema = load_alert_schema()?;
+
+    let command_interval: i64 = 500;
+    let mut command_check_countdown = command_interval;
+
+    loop {
+        if command_check_countdown == 0 {
+            match receiver.try_recv() {
+                Ok(WorkerCmd::TERM) => {
+                    info!("filterworker {} received termination command", &id);
+                    break;
+                }
+                Err(TryRecvError::Disconnected) => {
+                    warn!("filter worker {} receiver disconnected, terminating", &id);
+                    break;
+                }
+                Err(TryRecvError::Empty) => {
+                    command_check_countdown = command_interval;
+                }
+            }
+        }
+        // if the queue is empty, wait for a bit and continue the loop
+        let queue_len: i64 = con
+            .llen(&input_queue)
+            .await
+            .map_err(FilterWorkerError::ConnectRedisError)?;
+        if queue_len == 0 {
+            tokio::time::sleep(std::time::Duration::from_millis(500)).await;
+            continue;
+        }
+
+        // get candids from redis
+        let alerts: Vec<String> = con
+            .rpop::<&str, Vec<String>>(&input_queue, NonZero::new(1000))
+            .await
+            .map_err(FilterWorkerError::PopCandidError)?;
+
+        let nb_alerts = alerts.len();
+        if nb_alerts == 0 {
+            // sleep for a bit if no alerts were found
+            tokio::time::sleep(tokio::time::Duration::from_secs(1)).await;
+            continue;
+        }
+
+        let alerts_output = filter_worker.process_alerts(&alerts).await?;
+        for alert in alerts_output {
+            send_alert_to_kafka(&alert, &schema, &producer, &output_topic, &id).await?;
+            trace!(
+                "Sent alert with candid {} to Kafka topic {}",
+                &alert.candid,
+                &output_topic
+            );
+        }
+        command_check_countdown -= nb_alerts as i64;
+    }
+
     Ok(())
 }
diff --git a/src/filter/lsst.rs b/src/filter/lsst.rs
index 9aa7dfbc..186ac5f4 100644
--- a/src/filter/lsst.rs
+++ b/src/filter/lsst.rs
@@ -1,14 +1,12 @@
+use futures::stream::StreamExt;
 use mongodb::bson::{doc, Document};
-use redis::AsyncCommands;
-use std::{collections::HashMap, num::NonZero};
-use tokio::sync::mpsc;
-use tokio::sync::mpsc::error::TryRecvError;
-use tracing::{info, warn};
+use std::collections::HashMap;
+use tracing::info;
 
 use crate::filter::{
-    get_filter_object, process_alerts, Filter, FilterError, FilterWorker, FilterWorkerError,
+    get_filter_object, run_filter, Alert, Filter, FilterError, FilterResults, FilterWorker,
+    FilterWorkerError, Origin, Photometry, Survey,
 };
-use crate::utils::worker::WorkerCmd;
 
 pub struct LsstFilter {
     id: i32,
@@ -117,36 +115,24 @@ impl Filter for LsstFilter {
 }
 
 pub struct LsstFilterWorker {
-    id: String,
-    receiver: mpsc::Receiver<WorkerCmd>,
-    filter_collection: mongodb::Collection<mongodb::bson::Document>,
     alert_collection: mongodb::Collection<mongodb::bson::Document>,
+    input_queue: String,
+    output_topic: String,
+    filters: Vec<LsstFilter>,
 }
 
 #[async_trait::async_trait]
 impl FilterWorker for LsstFilterWorker {
-    async fn new(
-        id: String,
-        receiver: mpsc::Receiver<WorkerCmd>,
-        config_path: &str,
-    ) -> Result<Self, FilterWorkerError> {
+    async fn new(config_path: &str) -> Result<Self, FilterWorkerError> {
         let config_file = crate::conf::load_config(&config_path)?;
         let db: mongodb::Database = crate::conf::build_db(&config_file).await?;
         let alert_collection = db.collection("LSST_alerts");
         let filter_collection = db.collection("filters");
 
-        Ok(LsstFilterWorker {
-            id,
-            receiver,
-            filter_collection,
-            alert_collection,
-        })
-    }
+        let input_queue = "LSST_alerts_filter_queue".to_string();
+        let output_topic = "LSST_alerts_results".to_string();
 
-    async fn run(&mut self) -> Result<(), FilterWorkerError> {
-        // query the DB to find the ids of all the filters for LSST that are active
-        let filter_ids: Vec<i32> = self
-            .filter_collection
+        let filter_ids: Vec<i32> = filter_collection
             .distinct("filter_id", doc! {"active": true, "catalog": "LSST_alerts"})
             .await
             .map_err(FilterWorkerError::GetFiltersError)?
@@ -157,105 +143,259 @@ impl FilterWorker for LsstFilterWorker {
 
         let mut filters: Vec<LsstFilter> = Vec::new();
         for filter_id in filter_ids {
-            filters.push(LsstFilter::build(filter_id, &self.filter_collection).await?);
+            filters.push(LsstFilter::build(filter_id, &filter_collection).await?);
         }
 
-        if filters.is_empty() {
-            warn!("no filters found for LSST");
-            return Ok(());
-        }
+        Ok(LsstFilterWorker {
+            alert_collection,
+            input_queue,
+            output_topic,
+            filters,
+        })
+    }
 
-        // LSST is simpler, there are no permissions so there is only one queue
-        let queue_name = "LSST_alerts_filter_queue";
-        // create a list of output queues, one for each filter
-        let mut filter_results_queues: HashMap<i32, String> = HashMap::new();
-        for filter in &filters {
-            let queue_name = format!("LSST_alerts_filter_{}_results_queue", filter.id);
-            filter_results_queues.insert(filter.id, queue_name);
-        }
+    fn input_queue_name(&self) -> String {
+        self.input_queue.clone()
+    }
 
-        // in a never ending loop, get candids from redis
-        let client_redis = redis::Client::open("redis://localhost:6379".to_string())
-            .map_err(FilterWorkerError::ConnectRedisError)?;
-        let mut con = client_redis
-            .get_multiplexed_async_connection()
-            .await
-            .map_err(FilterWorkerError::ConnectRedisError)?;
+    fn output_topic_name(&self) -> String {
+        self.output_topic.clone()
+    }
 
-        let command_interval: i64 = 500;
-        let mut command_check_countdown = command_interval;
+    fn has_filters(&self) -> bool {
+        !self.filters.is_empty()
+    }
 
-        loop {
-            if command_check_countdown == 0 {
-                match self.receiver.try_recv() {
-                    Ok(WorkerCmd::TERM) => {
-                        info!("filterworker {} received termination command", self.id);
-                        break;
-                    }
-                    Err(TryRecvError::Disconnected) => {
-                        warn!(
-                            "filter worker {} receiver disconnected, terminating",
-                            self.id
-                        );
-                        break;
-                    }
-                    Err(TryRecvError::Empty) => {
-                        command_check_countdown = command_interval;
+    async fn build_alert(
+        &self,
+        candid: i64,
+        filter_results: Vec<FilterResults>,
+    ) -> Result<Alert, FilterWorkerError> {
+        let pipeline = vec![
+            doc! {
+                "$match": {
+                    "_id": candid
+                }
+            },
+            doc! {
+                "$project": {
+                    "objectId": 1,
+                    "jd": "$candidate.jd",
+                    "ra": "$candidate.ra",
+                    "dec": "$candidate.dec",
+                    "cutoutScience": 1,
+                    "cutoutTemplate": 1,
+                    "cutoutDifference": 1
+                }
+            },
+            doc! {
+                "$lookup": {
+                    "from": "LSST_alerts_aux",
+                    "localField": "objectId",
+                    "foreignField": "_id",
+                    "as": "aux"
+                }
+            },
+            doc! {
+                "$lookup": {
+                    "from": "LSST_alerts_cutouts",
+                    "localField": "_id",
+                    "foreignField": "_id",
+                    "as": "cutouts"
+                }
+            },
+            doc! {
+                "$project": {
+                    "objectId": 1,
+                    "jd": 1,
+                    "ra": 1,
+                    "dec": 1,
+                    "prv_candidates": {
+                        "$arrayElemAt": [
+                            "$aux.prv_candidates",
+                            0
+                        ]
+                    },
+                    "prv_nondetections": {
+                        "$arrayElemAt": [
+                            "$aux.prv_nondetections",
+                            0
+                        ]
+                    },
+                    "cutoutScience": {
+                        "$arrayElemAt": [
+                            "$cutouts.cutoutScience",
+                            0
+                        ]
+                    },
+                    "cutoutTemplate": {
+                        "$arrayElemAt": [
+                            "$cutouts.cutoutTemplate",
+                            0
+                        ]
+                    },
+                    "cutoutDifference": {
+                        "$arrayElemAt": [
+                            "$cutouts.cutoutDifference",
+                            0
+                        ]
                     }
                 }
-            }
-            // if the queue is empty, wait for a bit and continue the loop
-            let queue_len: i64 = con
-                .llen(queue_name)
-                .await
-                .map_err(FilterWorkerError::ConnectRedisError)?;
-            if queue_len == 0 {
-                tokio::time::sleep(std::time::Duration::from_millis(500)).await;
-                continue;
-            }
-            // get candids from redis
-            let candids: Vec<i64> = con
-                .rpop::<&str, Vec<i64>>(queue_name, NonZero::new(1000))
-                .await
-                .map_err(FilterWorkerError::PopCandidError)?;
-
-            let nb_candids = candids.len();
-            if nb_candids == 0 {
+            },
+        ];
+
+        // Execute the aggregation pipeline
+        let mut cursor = self
+            .alert_collection
+            .aggregate(pipeline)
+            .await
+            .map_err(FilterWorkerError::GetAlertByCandidError)?;
+
+        let alert_document = if let Some(result) = cursor.next().await {
+            result.map_err(FilterWorkerError::GetAlertByCandidError)?
+        } else {
+            return Err(FilterWorkerError::AlertNotFound);
+        };
+
+        let object_id = alert_document.get_i64("objectId")?;
+        let jd = alert_document.get_f64("jd")?;
+        let ra = alert_document.get_f64("ra")?;
+        let dec = alert_document.get_f64("dec")?;
+        let cutout_science = alert_document.get_binary_generic("cutoutScience")?.to_vec();
+        let cutout_template = alert_document
+            .get_binary_generic("cutoutTemplate")?
+            .to_vec();
+        let cutout_difference = alert_document
+            .get_binary_generic("cutoutDifference")?
+            .to_vec();
+
+        // let's create the array of photometry (non forced phot only for now)
+        let mut photometry = Vec::new();
+        for doc in alert_document.get_array("prv_candidates")?.iter() {
+            let doc = match doc.as_document() {
+                Some(doc) => doc,
+                None => continue, // skip if not a document
+            };
+            let jd = doc.get_f64("jd")?;
+            let flux = doc.get_f64("psfFlux")?;
+            let flux_err = doc.get_f64("psfFluxErr")?;
+            let band = doc.get_str("band")?.to_string();
+            let ra = doc.get_f64("ra").ok(); // optional, might not be present
+            let dec = doc.get_f64("dec").ok(); // optional, might not be present
+
+            photometry.push(Photometry {
+                jd,
+                flux: Some(flux),
+                flux_err,
+                band: format!("lsst{}", band),
+                zero_point: 8.9,
+                origin: Origin::Alert,
+                programid: 1, // only one public stream for LSST
+                survey: Survey::LSST,
+                ra,
+                dec,
+            });
+        }
+
+        // next we do the non detections
+        for doc in alert_document.get_array("prv_nondetections")?.iter() {
+            let doc = match doc.as_document() {
+                Some(doc) => doc,
+                None => continue, // skip if not a document
+            };
+            let jd = doc.get_f64("jd")?;
+            let flux_err = doc.get_f64("noise")?;
+            let band = doc.get_str("band")?.to_string();
+
+            photometry.push(Photometry {
+                jd,
+                flux: None, // for non-detections, flux is None
+                flux_err,
+                band: format!("lsst{}", band),
+                zero_point: 8.9,
+                origin: Origin::Alert,
+                programid: 1, // only one public stream for LSST
+                survey: Survey::LSST,
+                ra: None,
+                dec: None,
+            });
+        }
+
+        // we ignore the forced photometry for now, but will add it later
+
+        let alert = Alert {
+            candid,
+            object_id: format!("{}", object_id),
+            jd,
+            ra,
+            dec,
+            filters: filter_results, // assuming you have filter results to attach
+            photometry,
+            cutout_science,
+            cutout_template,
+            cutout_difference,
+        };
+
+        Ok(alert)
+    }
+
+    async fn process_alerts(&mut self, alerts: &[String]) -> Result<Vec<Alert>, FilterWorkerError> {
+        let mut alerts_output = Vec::new();
+
+        // unlike ZTF where we get a tuple of (programid, candid) from redis
+        // LSST has only one public stream, meaning there are no programids
+        // so we simply convert the array of String to Vec<i64>
+        let candids: Vec<i64> = alerts.iter().map(|alert| alert.parse().unwrap()).collect();
+
+        // run the filters
+        let mut results_map: HashMap<i64, Vec<FilterResults>> = HashMap::new();
+        for filter in &self.filters {
+            let out_documents = run_filter(
+                candids.clone(),
+                filter.pipeline.clone(),
+                &self.alert_collection,
+            )
+            .await?;
+
+            // if the array is empty, continue
+            if out_documents.is_empty() {
                 continue;
+            } else {
+                // if we have output documents, we need to process them
+                // and create filter results for each document (which contain annotations)
+                info!(
+                    "{} alerts passed lsst filter {}",
+                    out_documents.len(),
+                    filter.id,
+                );
             }
-            // run the filters
-            for filter in &filters {
-                let out_documents = process_alerts(
-                    candids.clone(),
-                    filter.pipeline.clone(),
-                    &self.alert_collection,
-                )
-                .await?;
-                // convert the documents to json
-                let out_documents: Vec<String> = out_documents
-                    .iter()
-                    .map(|x| {
-                        serde_json::to_string(x)
-                            .map_err(FilterWorkerError::SerializeFilterResultError)
-                    })
-                    .filter_map(Result::ok)
-                    .collect();
-                // push results to redis
-                if out_documents.is_empty() {
-                    continue;
-                }
 
-                let queue_name = filter_results_queues
-                    .get(&filter.id)
-                    .ok_or(FilterWorkerError::GetQueueNameError)?;
+            let now_ts = chrono::Utc::now().timestamp_millis() as f64;
 
-                con.lpush::<_, _, ()>(queue_name, out_documents)
-                    .await
-                    .map_err(FilterWorkerError::PushFilterResultsError)?;
+            for doc in out_documents {
+                let candid = doc.get_i64("_id")?;
+                // might want to have the annotations as an optional field instead of empty
+                let annotations =
+                    serde_json::to_string(doc.get_document("annotations").unwrap_or(&doc! {}))
+                        .map_err(FilterWorkerError::SerializeFilterResultError)?;
+                let filter_result = FilterResults {
+                    filter_id: filter.id,
+                    passed_at: now_ts,
+                    annotations,
+                };
+                let entry = results_map.entry(candid).or_insert(Vec::new());
+                entry.push(filter_result);
             }
-            command_check_countdown -= nb_candids as i64;
         }
 
-        Ok(())
+        // now we've basically combined the filter results for each candid
+        // we build the alert output and send it to Kafka
+        for (candid, filter_results) in &results_map {
+            let alert = self.build_alert(*candid, filter_results.clone()).await?;
+
+            alerts_output.push(alert);
+        }
+
+        Ok(alerts_output)
     }
 }
diff --git a/src/filter/mod.rs b/src/filter/mod.rs
index f41719e3..a67fe09b 100644
--- a/src/filter/mod.rs
+++ b/src/filter/mod.rs
@@ -2,9 +2,12 @@ mod base;
 mod lsst;
 mod ztf;
 
-use base::get_filter_object;
+use base::{
+    get_filter_object, parse_programid_candid_tuple, Alert, FilterResults, Origin, Photometry,
+    Survey,
+};
 pub use base::{
-    process_alerts, run_filter_worker, Filter, FilterError, FilterWorker, FilterWorkerError,
+    run_filter, run_filter_worker, Filter, FilterError, FilterWorker, FilterWorkerError,
 };
 pub use lsst::{LsstFilter, LsstFilterWorker};
 pub use ztf::{ZtfFilter, ZtfFilterWorker};
diff --git a/src/filter/ztf.rs b/src/filter/ztf.rs
index 259a6058..c8ebb8e0 100644
--- a/src/filter/ztf.rs
+++ b/src/filter/ztf.rs
@@ -1,14 +1,13 @@
+use flare::phot::{limmag_to_fluxerr, mag_to_flux};
+use futures::stream::StreamExt;
 use mongodb::bson::{doc, Document};
-use redis::AsyncCommands;
-use std::{collections::HashMap, num::NonZero};
-use tokio::sync::mpsc;
-use tokio::sync::mpsc::error::TryRecvError;
+use std::collections::HashMap;
 use tracing::{info, warn};
 
 use crate::filter::{
-    get_filter_object, process_alerts, Filter, FilterError, FilterWorker, FilterWorkerError,
+    get_filter_object, parse_programid_candid_tuple, run_filter, Alert, Filter, FilterError,
+    FilterResults, FilterWorker, FilterWorkerError, Origin, Photometry, Survey,
 };
-use crate::utils::worker::WorkerCmd;
 
 #[derive(Debug)]
 pub struct ZtfFilter {
@@ -147,35 +146,25 @@ impl Filter for ZtfFilter {
 }
 
 pub struct ZtfFilterWorker {
-    id: String,
-    receiver: mpsc::Receiver<WorkerCmd>,
-    filter_collection: mongodb::Collection<mongodb::bson::Document>,
     alert_collection: mongodb::Collection<mongodb::bson::Document>,
+    input_queue: String,
+    output_topic: String,
+    filters: Vec<ZtfFilter>,
+    filters_by_permission: HashMap<i32, Vec<usize>>,
 }
 
 #[async_trait::async_trait]
 impl FilterWorker for ZtfFilterWorker {
-    async fn new(
-        id: String,
-        receiver: mpsc::Receiver<WorkerCmd>,
-        config_path: &str,
-    ) -> Result<Self, FilterWorkerError> {
+    async fn new(config_path: &str) -> Result<Self, FilterWorkerError> {
         let config_file = crate::conf::load_config(&config_path)?;
         let db: mongodb::Database = crate::conf::build_db(&config_file).await?;
         let alert_collection = db.collection("ZTF_alerts");
         let filter_collection = db.collection("filters");
-        Ok(ZtfFilterWorker {
-            id,
-            receiver,
-            filter_collection,
-            alert_collection,
-        })
-    }
 
-    async fn run(&mut self) -> Result<(), FilterWorkerError> {
-        // query the DB to find the ids of all the filters for ZTF that are active
-        let filter_ids: Vec<i32> = self
-            .filter_collection
+        let input_queue = "ZTF_alerts_filter_queue".to_string();
+        let output_topic = "ZTF_alerts_results".to_string();
+
+        let filter_ids: Vec<i32> = filter_collection
             .distinct("filter_id", doc! {"active": true, "catalog": "ZTF_alerts"})
             .await
             .map_err(FilterWorkerError::GetFiltersError)?
@@ -186,26 +175,9 @@ impl FilterWorker for ZtfFilterWorker {
 
         let mut filters: Vec<ZtfFilter> = Vec::new();
         for filter_id in filter_ids {
-            filters.push(ZtfFilter::build(filter_id, &self.filter_collection).await?);
+            filters.push(ZtfFilter::build(filter_id, &filter_collection).await?);
         }
 
-        if filters.is_empty() {
-            warn!("no filters found for ZTF");
-            return Ok(());
-        }
-
-        // get the highest permissions accross all filters
-        let mut max_permission = 0;
-        for filter in &filters {
-            for permission in &filter.permissions {
-                if *permission > max_permission {
-                    max_permission = *permission;
-                }
-            }
-        }
-        // create a list of queues to read from
-        let input_queue = "ZTF_alerts_filter_queue".to_string();
-
         // create a hashmap of filters per programid (permissions)
         // basically we'll have the 4 programid (from 0 to 3) as keys
         // and the idx of the filters that have that programid in their permissions as values
@@ -219,109 +191,283 @@ impl FilterWorker for ZtfFilterWorker {
             }
         }
 
-        // create a list of output queues, one for each filter
-        let mut filter_results_queues: HashMap<i32, String> = HashMap::new();
-        for filter in &filters {
-            let queue_name = format!("ZTF_alerts_filter_{}_results_queue", filter.id);
-            filter_results_queues.insert(filter.id, queue_name);
-        }
+        Ok(ZtfFilterWorker {
+            alert_collection,
+            input_queue,
+            output_topic,
+            filters,
+            filters_by_permission,
+        })
+    }
 
-        // in a never ending loop, loop over the queues
-        let client_redis = redis::Client::open("redis://localhost:6379".to_string())
-            .map_err(FilterWorkerError::ConnectRedisError)?;
-        let mut con = client_redis
-            .get_multiplexed_async_connection()
-            .await
-            .map_err(FilterWorkerError::ConnectRedisError)?;
+    fn input_queue_name(&self) -> String {
+        self.input_queue.clone()
+    }
 
-        let command_interval: i64 = 500;
-        let mut command_check_countdown = command_interval;
+    fn output_topic_name(&self) -> String {
+        self.output_topic.clone()
+    }
 
-        loop {
-            if command_check_countdown == 0 {
-                match self.receiver.try_recv() {
-                    Ok(WorkerCmd::TERM) => {
-                        info!("filterworker {} received termination command", self.id);
-                        break;
-                    }
-                    Err(TryRecvError::Disconnected) => {
-                        warn!(
-                            "filter worker {} receiver disconnected, terminating",
-                            self.id
-                        );
-                        break;
-                    }
-                    Err(TryRecvError::Empty) => {
-                        command_check_countdown = command_interval;
+    fn has_filters(&self) -> bool {
+        !self.filters.is_empty()
+    }
+
+    async fn build_alert(
+        &self,
+        candid: i64,
+        filter_results: Vec<FilterResults>,
+    ) -> Result<Alert, FilterWorkerError> {
+        let pipeline = vec![
+            doc! {
+                "$match": {
+                    "_id": candid
+                }
+            },
+            doc! {
+                "$project": {
+                    "objectId": 1,
+                    "jd": "$candidate.jd",
+                    "ra": "$candidate.ra",
+                    "dec": "$candidate.dec",
+                    "cutoutScience": 1,
+                    "cutoutTemplate": 1,
+                    "cutoutDifference": 1
+                }
+            },
+            doc! {
+                "$lookup": {
+                    "from": "ZTF_alerts_aux",
+                    "localField": "objectId",
+                    "foreignField": "_id",
+                    "as": "aux"
+                }
+            },
+            doc! {
+                "$lookup": {
+                    "from": "ZTF_alerts_cutouts",
+                    "localField": "_id",
+                    "foreignField": "_id",
+                    "as": "cutouts"
+                }
+            },
+            doc! {
+                "$project": {
+                    "objectId": 1,
+                    "jd": 1,
+                    "ra": 1,
+                    "dec": 1,
+                    "prv_candidates": {
+                        "$arrayElemAt": [
+                            "$aux.prv_candidates",
+                            0
+                        ]
+                    },
+                    "prv_nondetections": {
+                        "$arrayElemAt": [
+                            "$aux.prv_nondetections",
+                            0
+                        ]
+                    },
+                    "cutoutScience": {
+                        "$arrayElemAt": [
+                            "$cutouts.cutoutScience",
+                            0
+                        ]
+                    },
+                    "cutoutTemplate": {
+                        "$arrayElemAt": [
+                            "$cutouts.cutoutTemplate",
+                            0
+                        ]
+                    },
+                    "cutoutDifference": {
+                        "$arrayElemAt": [
+                            "$cutouts.cutoutDifference",
+                            0
+                        ]
                     }
                 }
-            }
-            // get candids from redis
-            let alerts: Vec<(i64, i32)> = con
-                .rpop::<&str, Vec<(i64, i32)>>(&input_queue, NonZero::new(1000))
-                .await
-                .map_err(FilterWorkerError::PopCandidError)?;
-
-            let nb_alerts = alerts.len();
-            if nb_alerts == 0 {
-                // sleep for a bit if no alerts were found
-                tokio::time::sleep(tokio::time::Duration::from_secs(1)).await;
-                continue;
-            }
+            },
+        ];
+
+        // Execute the aggregation pipeline
+        let mut cursor = self
+            .alert_collection
+            .aggregate(pipeline)
+            .await
+            .map_err(FilterWorkerError::GetAlertByCandidError)?;
+
+        let alert_document = if let Some(result) = cursor.next().await {
+            result.map_err(FilterWorkerError::GetAlertByCandidError)?
+        } else {
+            return Err(FilterWorkerError::AlertNotFound);
+        };
+
+        let object_id = alert_document.get_str("objectId")?.to_string();
+        let jd = alert_document.get_f64("jd")?;
+        let ra = alert_document.get_f64("ra")?;
+        let dec = alert_document.get_f64("dec")?;
+        let cutout_science = alert_document.get_binary_generic("cutoutScience")?.to_vec();
+        let cutout_template = alert_document
+            .get_binary_generic("cutoutTemplate")?
+            .to_vec();
+        let cutout_difference = alert_document
+            .get_binary_generic("cutoutDifference")?
+            .to_vec();
 
-            // retrieve alerts to process and group by programid
-            let mut alerts_by_programid: HashMap<i32, Vec<i64>> = HashMap::new();
-            for (candid, programid) in alerts {
-                let entry = alerts_by_programid.entry(programid).or_insert(Vec::new());
-                entry.push(candid);
+        // let's create the array of photometry (non forced phot only for now)
+        let mut photometry = Vec::new();
+        for doc in alert_document.get_array("prv_candidates")?.iter() {
+            let doc = match doc.as_document() {
+                Some(doc) => doc,
+                None => continue, // skip if not a document
+            };
+            let jd = doc.get_f64("jd")?;
+            let mag = doc.get_f64("magpsf")?;
+            let mag_err = doc.get_f64("sigmapsf")?;
+            let isdiffpos = doc.get_bool("isdiffpos")?;
+            let band = doc.get_str("band")?.to_string();
+            let programid = doc.get_i32("programid")?;
+            let zero_point = 23.9;
+            let ra = doc.get_f64("ra").ok(); // optional, might not be present
+            let dec = doc.get_f64("dec").ok(); // optional, might not be present
+
+            let (flux, flux_err) = mag_to_flux(mag, mag_err, zero_point);
+            photometry.push(Photometry {
+                jd,
+                flux: match isdiffpos {
+                    true => Some(flux),
+                    false => Some(-1.0 * flux),
+                },
+                flux_err,
+                band: format!("ztf{}", band),
+                zero_point,
+                origin: Origin::Alert,
+                programid,
+                survey: Survey::ZTF,
+                ra,
+                dec,
+            });
+        }
+
+        // next we do the non detections
+        for doc in alert_document.get_array("prv_nondetections")?.iter() {
+            let doc = match doc.as_document() {
+                Some(doc) => doc,
+                None => continue, // skip if not a document
+            };
+            let jd = doc.get_f64("jd")?;
+            let mag_limit = doc.get_f64("diffmaglim")?;
+            let band = doc.get_str("band")?.to_string();
+            let programid = doc.get_i32("programid")?;
+            let zero_point = 23.9;
+
+            let flux_err = limmag_to_fluxerr(mag_limit, zero_point, 5.0);
+
+            photometry.push(Photometry {
+                jd,
+                flux: None, // for non-detections, flux is None
+                flux_err,
+                band: format!("ztf{}", band),
+                zero_point,
+                origin: Origin::Alert,
+                programid,
+                survey: Survey::ZTF,
+                ra: None,
+                dec: None,
+            });
+        }
+
+        // we ignore the forced photometry for now, but will add it later
+
+        let alert = Alert {
+            candid,
+            object_id,
+            jd,
+            ra,
+            dec,
+            filters: filter_results, // assuming you have filter results to attach
+            photometry,
+            cutout_science,
+            cutout_template,
+            cutout_difference,
+        };
+
+        Ok(alert)
+    }
+
+    async fn process_alerts(&mut self, alerts: &[String]) -> Result<Vec<Alert>, FilterWorkerError> {
+        let mut alerts_output = Vec::new();
+
+        // retrieve alerts to process and group by programid
+        let mut alerts_by_programid: HashMap<i32, Vec<i64>> = HashMap::new();
+        for tuple_str in alerts {
+            if let Some(tuple) = parse_programid_candid_tuple(&tuple_str) {
+                let entry = alerts_by_programid.entry(tuple.0).or_insert(Vec::new());
+                entry.push(tuple.1);
+            } else {
+                warn!("Failed to parse tuple from string: {}", tuple_str);
             }
+        }
 
-            // for each programid, get the filters that have that programid in their permissions
-            // and run the filters
-            for (programid, candids) in alerts_by_programid {
-                let filter_indices = filters_by_permission
-                    .get(&programid)
-                    .ok_or(FilterWorkerError::GetFilterByQueueError)?;
-
-                for i in filter_indices {
-                    let filter = &filters[*i];
-                    let out_documents = process_alerts(
-                        candids.clone(),
-                        filter.pipeline.clone(),
-                        &self.alert_collection,
-                    )
-                    .await?;
-
-                    // if the array is empty, continue
-                    if out_documents.is_empty() {
-                        continue;
-                    }
-                    // convert the documents to json
-                    let out_documents: Vec<String> = out_documents
-                        .iter()
-                        .map(|x| {
-                            serde_json::to_string(x)
-                                .map_err(FilterWorkerError::SerializeFilterResultError)
-                        })
-                        .filter_map(Result::ok)
-                        .collect();
-                    // push results to redis
-                    let queue_name = filter_results_queues
-                        .get(&filter.id)
-                        .ok_or(FilterWorkerError::GetQueueNameError)?;
-                    // push them all at once
-                    let _: () = match con.lpush(queue_name, out_documents).await {
-                        Ok(x) => x,
-                        Err(e) => {
-                            warn!("error pushing filter results to redis: {}", e);
-                            continue;
-                        }
+        // for each programid, get the filters that have that programid in their permissions
+        // and run the filters
+        for (programid, candids) in alerts_by_programid {
+            let mut results_map: HashMap<i64, Vec<FilterResults>> = HashMap::new();
+
+            let filter_indices = self
+                .filters_by_permission
+                .get(&programid)
+                .ok_or(FilterWorkerError::GetFilterByQueueError)?;
+
+            for i in filter_indices {
+                let filter = &self.filters[*i];
+                let out_documents = run_filter(
+                    candids.clone(),
+                    filter.pipeline.clone(),
+                    &self.alert_collection,
+                )
+                .await?;
+
+                // if the array is empty, continue
+                if out_documents.is_empty() {
+                    continue;
+                } else {
+                    // if we have output documents, we need to process them
+                    // and create filter results for each document (which contain annotations)
+                    info!(
+                        "{} alerts passed ztf filter {} with programid {}",
+                        out_documents.len(),
+                        filter.id,
+                        programid,
+                    );
+                }
+
+                let now_ts = chrono::Utc::now().timestamp_millis() as f64;
+
+                for doc in out_documents {
+                    let candid = doc.get_i64("_id")?;
+                    // might want to have the annotations as an optional field instead of empty
+                    let annotations =
+                        serde_json::to_string(doc.get_document("annotations").unwrap_or(&doc! {}))
+                            .map_err(FilterWorkerError::SerializeFilterResultError)?;
+                    let filter_result = FilterResults {
+                        filter_id: filter.id,
+                        passed_at: now_ts,
+                        annotations,
                     };
+                    let entry = results_map.entry(candid).or_insert(Vec::new());
+                    entry.push(filter_result);
                 }
             }
-            command_check_countdown -= nb_alerts as i64;
+
+            // now we've basically combined the filter results for each candid
+            for (candid, filter_results) in &results_map {
+                let alert = self.build_alert(*candid, filter_results.clone()).await?;
+                alerts_output.push(alert);
+            }
         }
 
-        Ok(())
+        Ok(alerts_output)
     }
 }
diff --git a/src/ml/base.rs b/src/ml/base.rs
index 9e4d8f04..6173a809 100644
--- a/src/ml/base.rs
+++ b/src/ml/base.rs
@@ -1,6 +1,6 @@
 use crate::{
     conf,
-    utils::fits::prepare_triplet,
+    utils::fits::{prepare_triplet, CutoutError},
     utils::worker::{get_check_command_interval, WorkerCmd},
 };
 use core::time;
@@ -14,11 +14,21 @@ use tracing::{info, warn};
 #[derive(thiserror::Error, Debug)]
 pub enum MLWorkerError {
     #[error("failed to connect to database")]
-    ConnectMongoError(#[from] mongodb::error::Error),
+    ConnectMongoError(#[source] mongodb::error::Error),
+    #[error("failed to retrieve candidates from database")]
+    RetrieveCandidatesDataError(#[source] mongodb::error::Error),
     #[error("failed to connect to redis")]
-    ConnectRedisError(#[from] redis::RedisError),
+    ConnectRedisError(#[source] redis::RedisError),
+    #[error("failed to retrieve candidates from redis")]
+    RedisError(#[source] redis::RedisError),
+    #[error("failed to push candidates to redis")]
+    RedisPushError(#[source] redis::RedisError),
     #[error("failed to read config")]
     ReadConfigError(#[from] conf::BoomConfigError),
+    #[error("could not find document field")]
+    MissingFieldError(#[from] mongodb::bson::document::ValueAccessError),
+    #[error("could not access cutout images")]
+    CutoutAccessError(#[from] CutoutError),
 }
 
 // fake ml worker which for now does not run any models
@@ -28,14 +38,15 @@ pub async fn run_ml_worker(
     mut receiver: mpsc::Receiver<WorkerCmd>,
     stream_name: &str,
     config_path: &str,
-) -> Result<(), Box<dyn std::error::Error>> {
+) -> Result<(), MLWorkerError> {
     let catalog: String = stream_name.to_string();
     let queue = format!("{}_alerts_classifier_queue", catalog);
     let output_queue = format!("{}_alerts_filter_queue", stream_name);
 
-    let config_file = conf::load_config(&config_path).unwrap();
+    let config_file = conf::load_config(&config_path)?;
     let db = conf::build_db(&config_file).await?;
-    let client_redis = redis::Client::open("redis://localhost:6379".to_string()).unwrap();
+    let client_redis = redis::Client::open("redis://localhost:6379".to_string())
+        .map_err(MLWorkerError::ConnectRedisError)?;
     let mut con = client_redis
         .get_multiplexed_async_connection()
         .await
@@ -61,7 +72,7 @@ pub async fn run_ml_worker(
         let candids = con
             .rpop::<&str, Vec<i64>>(queue.as_str(), NonZero::new(1000))
             .await
-            .unwrap();
+            .map_err(MLWorkerError::RedisError)?;
 
         let mut alert_cursor = db
             .collection::<Document>(format!("{}_alerts", catalog).as_str())
@@ -157,7 +168,7 @@ pub async fn run_ml_worker(
                 },
             ])
             .await
-            .unwrap();
+            .map_err(MLWorkerError::RetrieveCandidatesDataError)?;
 
         let mut alerts: Vec<Document> = Vec::new();
         while let Some(result) = alert_cursor.next().await {
@@ -194,28 +205,30 @@ pub async fn run_ml_worker(
         // run the models on batches of alerts instead of one by one
         let mut processed_candids = Vec::new();
         for alert in &alerts {
-            let candid = alert.get_i64("_id").unwrap();
-            let programid = alert
-                .get_document("candidate")
-                .unwrap()
-                .get_i32("programid")
-                .unwrap();
-            let obj_id = alert.get_str("objectId").unwrap();
-            let result = prepare_triplet(&alert);
-            if result.is_err() {
-                warn!(
-                    "ML WORKER {}: error preparing triplet for alert {}",
-                    id, obj_id
-                );
-                continue;
-            }
-            let (_cutout_science, _cutout_template, _cutout_difference) = result.unwrap();
+            let candid = alert.get_i64("_id")?;
+            let programid = alert.get_document("candidate")?.get_i32("programid")?;
+            let obj_id = alert.get_str("objectId")?;
+
+            // redis needs tuples as strings
+            processed_candids.push(format!("{},{}", programid, candid)); // Store the programid and candid as a string tuple
 
-            processed_candids.push((candid, programid));
+            let (_cutout_science, _cutout_template, _cutout_difference) =
+                match prepare_triplet(&alert) {
+                    Ok((cutout_science, cutout_template, cutout_difference)) => {
+                        (cutout_science, cutout_template, cutout_difference)
+                    }
+                    Err(e) => {
+                        warn!(
+                            "ML WORKER {}: error preparing triplet for alert {}: {}",
+                            id, obj_id, e
+                        );
+                        continue; // Skip this alert if there was an error
+                    }
+                };
         }
 
-        con.lpush::<&str, Vec<(i64, i32)>, usize>(output_queue.as_str(), processed_candids)
+        con.lpush::<&str, Vec<String>, usize>(output_queue.as_str(), processed_candids)
             .await
-            .unwrap();
+            .map_err(MLWorkerError::RedisPushError)?;
     }
 }
diff --git a/src/utils/fits.rs b/src/utils/fits.rs
index 60f6575f..ead9f730 100644
--- a/src/utils/fits.rs
+++ b/src/utils/fits.rs
@@ -9,6 +9,18 @@ const FITS_HEADER_LEN: usize = 2880; // FITS headers are in blocks of 2880 bytes
 const NAXIS_STANDARD: usize = 63;
 const NB_PIXELS: usize = NAXIS_STANDARD * NAXIS_STANDARD;
 
+#[derive(thiserror::Error, Debug)]
+pub enum CutoutError {
+    #[error("failed to convert buffer to image")]
+    BufferToImageError,
+    #[error("failed to normalize image")]
+    NormalizeImageError,
+    #[error("failed to prepare cutout")]
+    PrepareCutoutError,
+    #[error("could not parse naxis1 or naxis2 from FITS header")]
+    ParseNAXISError(#[from] std::num::ParseIntError),
+}
+
 fn u8_to_f32_vec(v: &[u8]) -> Vec<f32> {
     v.chunks_exact(4)
         .map(TryInto::try_into)
@@ -18,7 +30,7 @@ fn u8_to_f32_vec(v: &[u8]) -> Vec<f32> {
 }
 
 /// Converts a buffer of bytes from a gzipped FITS file to a vector of flattened 2D image data
-pub fn buffer_to_image(buffer: &[u8]) -> Result<Vec<f32>, Box<dyn std::error::Error>> {
+pub fn buffer_to_image(buffer: &[u8]) -> Result<Vec<f32>, CutoutError> {
     let mut decoder = GzDecoder::new(buffer);
 
     let mut decompressed_data = Vec::new();
@@ -49,7 +61,7 @@ pub fn buffer_to_image(buffer: &[u8]) -> Result<Vec<f32>, Box<dyn std::error::Er
     }
     let naxis1 = String::from_utf8_lossy(&subset[naxis1_val_start..naxis1_val_end])
         .parse::<usize>()
-        .unwrap();
+        .map_err(CutoutError::ParseNAXISError)?; // Parse the value of NAXIS1 from the FITS header
 
     let mut naxis2_key_start = naxis1_val_end;
     let mut naxis2_val_start = 0;
@@ -74,7 +86,7 @@ pub fn buffer_to_image(buffer: &[u8]) -> Result<Vec<f32>, Box<dyn std::error::Er
     }
     let naxis2 = String::from_utf8_lossy(&subset[naxis2_val_start..naxis2_val_end])
         .parse::<usize>()
-        .unwrap();
+        .map_err(CutoutError::ParseNAXISError)?; // Parse the value of NAXIS2 from the FITS header
 
     let mut image_data = u8_to_f32_vec(
         &decompressed_data[FITS_HEADER_LEN..(FITS_HEADER_LEN + (naxis1 * naxis2 * 4) as usize)],
@@ -103,7 +115,7 @@ pub fn buffer_to_image(buffer: &[u8]) -> Result<Vec<f32>, Box<dyn std::error::Er
 }
 
 /// Normalizes a flattened 2D image
-pub fn normalize_image(image: Vec<f32>) -> Result<Vec<f32>, Box<dyn std::error::Error>> {
+pub fn normalize_image(image: Vec<f32>) -> Result<Vec<f32>, CutoutError> {
     // replace all NaNs with 0s and clamp all values to the range [f32::MIN, f32::MAX]
     let mut normalized: Vec<f32> = image
         .into_iter()
@@ -127,33 +139,33 @@ pub fn normalize_image(image: Vec<f32>) -> Result<Vec<f32>, Box<dyn std::error::
 /// Prepares a cutout image for ML models
 /// It reads the image from the alert document
 /// decompresses it, normalizes it and returns it as a flattened 2D array of floats
-fn prepare_cutout(cutout: &[u8]) -> Result<Vec<f32>, Box<dyn std::error::Error>> {
-    let cutout = buffer_to_image(cutout).unwrap();
-    let cutout = normalize_image(cutout).unwrap();
+fn prepare_cutout(cutout: &[u8]) -> Result<Vec<f32>, CutoutError> {
+    let cutout = buffer_to_image(cutout)?;
+    let cutout = normalize_image(cutout)?;
     Ok(cutout)
 }
 
 /// Prepares a triplet of cutouts for ML models
 pub fn prepare_triplet(
     alert_doc: &mongodb::bson::Document,
-) -> Result<(Vec<f32>, Vec<f32>, Vec<f32>), Box<dyn std::error::Error>> {
+) -> Result<(Vec<f32>, Vec<f32>, Vec<f32>), CutoutError> {
     let cutout_science = alert_doc
         .get_binary_generic("cutoutScience")
         .unwrap()
         .to_vec();
-    let cutout_science = prepare_cutout(&cutout_science).unwrap();
+    let cutout_science = prepare_cutout(&cutout_science)?;
 
     let cutout_template = alert_doc
         .get_binary_generic("cutoutTemplate")
         .unwrap()
         .to_vec();
-    let cutout_template = prepare_cutout(&cutout_template).unwrap();
+    let cutout_template = prepare_cutout(&cutout_template)?;
 
     let cutout_difference = alert_doc
         .get_binary_generic("cutoutDifference")
         .unwrap()
         .to_vec();
-    let cutout_difference = prepare_cutout(&cutout_difference).unwrap();
+    let cutout_difference = prepare_cutout(&cutout_difference)?;
 
     Ok((cutout_science, cutout_template, cutout_difference))
 }
diff --git a/src/utils/testing.rs b/src/utils/testing.rs
index e85c4c9d..08e5daa9 100644
--- a/src/utils/testing.rs
+++ b/src/utils/testing.rs
@@ -83,12 +83,13 @@ pub async fn drop_alert_from_collections(
     Ok(())
 }
 
-// insert a test filter with id -1 into the database
-pub async fn insert_test_filter() -> Result<(), Box<dyn std::error::Error>> {
+pub async fn insert_test_ztf_filter() -> Result<i32, Box<dyn std::error::Error>> {
+    // we randomize the filter id
+    let filter_id = rand::random::<i32>();
     let filter_obj: mongodb::bson::Document = doc! {
       "_id": mongodb::bson::oid::ObjectId::new(),
       "group_id": 41,
-      "filter_id": -1,
+      "filter_id": filter_id,
       "catalog": "ZTF_alerts",
       "permissions": [
         1
@@ -98,7 +99,7 @@ pub async fn insert_test_filter() -> Result<(), Box<dyn std::error::Error>> {
       "fv": [
         {
             "fid": "v2e0fs",
-            "pipeline": "[{\"$match\": {\"candidate.drb\": {\"$gt\": 0.5}, \"candidate.ndethist\": {\"$gt\": 1.0}, \"candidate.magpsf\": {\"$lte\": 18.5}}}]",
+            "pipeline": "[{\"$match\": {\"candidate.drb\": {\"$gt\": 0.5}, \"candidate.ndethist\": {\"$gt\": 1.0}, \"candidate.magpsf\": {\"$lte\": 18.5}}}, {\"$project\": {\"annotations.mag_now\": {\"$round\": [\"$candidate.magpsf\", 2]}}}]",
             "created_at": {
             "$date": "2020-10-21T08:39:43.693Z"
             }
@@ -127,16 +128,74 @@ pub async fn insert_test_filter() -> Result<(), Box<dyn std::error::Error>> {
         _ => {}
     }
 
+    Ok(filter_id)
+}
+
+pub async fn remove_test_ztf_filter(filter_id: i32) -> Result<(), Box<dyn std::error::Error>> {
+    let config_file = conf::load_config("tests/config.test.yaml")?;
+    let db = conf::build_db(&config_file).await?;
+    let _ = db
+        .collection::<mongodb::bson::Document>("filters")
+        .delete_many(doc! {"filter_id": filter_id, "catalog": "ZTF_alerts"})
+        .await;
+
     Ok(())
 }
 
-// remove test filter with id -1 from the database
-pub async fn remove_test_filter() -> Result<(), Box<dyn std::error::Error>> {
+pub async fn insert_test_lsst_filter() -> Result<i32, Box<dyn std::error::Error>> {
+    // we randomize the filter id
+    let filter_id = rand::random::<i32>();
+    let filter_obj: mongodb::bson::Document = doc! {
+      "_id": mongodb::bson::oid::ObjectId::new(),
+      "group_id": 41,
+      "filter_id": filter_id,
+      "catalog": "LSST_alerts",
+      "permissions": [
+        1
+      ],
+      "active": true,
+      "active_fid": "v2e0fs",
+      "fv": [
+        {
+            "fid": "v2e0fs",
+            "pipeline": "[{\"$match\": {\"candidate.reliability\": {\"$gt\": 0.5}, \"candidate.snr\": {\"$gt\": 5.0}, \"candidate.magpsf\": {\"$lte\": 25.0}}}, {\"$project\": {\"annotations.mag_now\": {\"$round\": [\"$candidate.magpsf\", 2]}}}]",
+            "created_at": {
+            "$date": "2020-10-21T08:39:43.693Z"
+            }
+        }
+      ],
+      "autosave": false,
+      "update_annotations": true,
+      "created_at": {
+        "$date": "2021-02-20T08:18:28.324Z"
+      },
+      "last_modified": {
+        "$date": "2023-05-04T23:39:07.090Z"
+      }
+    };
+
+    let config_file = conf::load_config("tests/config.test.yaml")?;
+    let db = conf::build_db(&config_file).await?;
+    let x = db
+        .collection::<mongodb::bson::Document>("filters")
+        .insert_one(filter_obj)
+        .await;
+    match x {
+        Err(e) => {
+            error!("error inserting filter obj: {}", e);
+        }
+        _ => {}
+    }
+
+    Ok(filter_id)
+}
+
+pub async fn remove_test_lsst_filter(filter_id: i32) -> Result<(), Box<dyn std::error::Error>> {
     let config_file = conf::load_config("tests/config.test.yaml")?;
     let db = conf::build_db(&config_file).await?;
     let _ = db
         .collection::<mongodb::bson::Document>("filters")
-        .delete_one(doc! {"filter_id": -1})
+        .delete_many(doc! {"filter_id": filter_id, "catalog": "LSST_alerts"})
         .await;
 
     Ok(())
@@ -350,7 +409,7 @@ impl AlertRandomizerTrait for LsstAlertRandomizer {
     type ObjectId = i64;
 
     fn default() -> Self {
-        let payload = fs::read("tests/data/alerts/lsst/0.avro").unwrap();
+        let payload = fs::read("tests/data/alerts/lsst/25409136044802067.avro").unwrap();
         Self {
             payload: Some(payload),
             schema_registry: SchemaRegistry::new(LSST_SCHEMA_REGISTRY_URL),
@@ -362,7 +421,7 @@ impl AlertRandomizerTrait for LsstAlertRandomizer {
     }
 
     fn new() -> Self {
-        let payload = fs::read("tests/data/alerts/lsst/0.avro").unwrap();
+        let payload = fs::read("tests/data/alerts/lsst/25409136044802067.avro").unwrap();
         Self {
             payload: Some(payload),
             schema_registry: SchemaRegistry::new(LSST_SCHEMA_REGISTRY_URL),
@@ -411,7 +470,7 @@ impl AlertRandomizerTrait for LsstAlertRandomizer {
         let dec = self.dec.unwrap_or_else(Self::randomize_dec);
         let payload = self
             .payload
-            .unwrap_or_else(|| fs::read("tests/data/alerts/lsst/0.avro").unwrap());
+            .unwrap_or_else(|| fs::read("tests/data/alerts/lsst/25409136044802067.avro").unwrap());
 
         let header = payload[0..5].to_vec();
 
diff --git a/tests/data/alerts/lsst/0.avro b/tests/data/alerts/lsst/25409136044802067.avro
similarity index 100%
rename from tests/data/alerts/lsst/0.avro
rename to tests/data/alerts/lsst/25409136044802067.avro
diff --git a/tests/test_filter.rs b/tests/test_filter.rs
index dbe2b31b..3f91f920 100644
--- a/tests/test_filter.rs
+++ b/tests/test_filter.rs
@@ -1,12 +1,9 @@
-use mongodb::bson::{doc, Document};
-use tracing::{error, info};
-
 use boom::{
-    alert::{AlertWorker, ZtfAlertWorker},
     conf,
-    filter::{process_alerts, Filter, ZtfFilter},
+    filter::{Filter, ZtfFilter},
     utils::testing,
 };
+use mongodb::bson::{doc, Document};
 
 const CONFIG_FILE: &str = "tests/config.test.yaml";
 
@@ -15,9 +12,11 @@ async fn test_build_filter() {
     let config_file = conf::load_config(CONFIG_FILE).unwrap();
     let db = conf::build_db(&config_file).await.unwrap();
     let filter_collection = db.collection("filters");
-    testing::insert_test_filter().await.unwrap();
-    let filter_result = ZtfFilter::build(-1, &filter_collection).await;
-    testing::remove_test_filter().await.unwrap();
+
+    let filter_id = testing::insert_test_ztf_filter().await.unwrap();
+    let filter_result = ZtfFilter::build(filter_id, &filter_collection).await;
+    testing::remove_test_ztf_filter(filter_id).await.unwrap();
+
     let filter = filter_result.unwrap();
     let pipeline: Vec<Document> = vec![
         doc! { "$match": {} },
@@ -42,6 +41,7 @@ async fn test_build_filter() {
             }
         },
         doc! { "$match": { "candidate.drb": { "$gt": 0.5 }, "candidate.ndethist": { "$gt": 1_f64 }, "candidate.magpsf": { "$lte": 18.5 } } },
+        doc! { "$project": { "annotations.mag_now": { "$round": ["$candidate.magpsf", 2_i64]} } },
     ];
     assert_eq!(pipeline, filter.pipeline);
     assert_eq!(vec![1], filter.permissions);
@@ -51,10 +51,10 @@ async fn test_build_filter() {
 async fn test_filter_found() {
     let config_file = conf::load_config("tests/config.test.yaml").unwrap();
     let db = conf::build_db(&config_file).await.unwrap();
-    testing::insert_test_filter().await.unwrap();
+    let filter_id = testing::insert_test_ztf_filter().await.unwrap();
     let filter_collection = db.collection("filters");
-    let filter_result = ZtfFilter::build(-1, &filter_collection).await;
-    testing::remove_test_filter().await.unwrap();
+    let filter_result = ZtfFilter::build(filter_id, &filter_collection).await;
+    testing::remove_test_ztf_filter(filter_id).await.unwrap();
     assert!(filter_result.is_ok());
 }
 
@@ -66,66 +66,3 @@ async fn test_no_filter_found() {
     let filter_result = ZtfFilter::build(-2, &filter_collection).await;
     assert!(filter_result.is_err());
 }
-
-// checks result of running filter
-#[tokio::test]
-async fn test_run_filter() {
-    testing::insert_test_filter().await.unwrap();
-    let config_file = conf::load_config(CONFIG_FILE).unwrap();
-    let db = conf::build_db(&config_file).await.unwrap();
-    let alert_collection = db.collection("ZTF_alerts");
-    let filter_collection = db.collection("filters");
-    let filter_result = ZtfFilter::build(-1, &filter_collection).await;
-    testing::remove_test_filter().await.unwrap();
-    assert!(filter_result.is_ok());
-    let filter = filter_result.unwrap();
-
-    let test_col_name = "ZTF_alerts";
-    let test_cutout_col_name = "ZTF_alerts_cutouts";
-    let test_aux_col_name = "ZTF_alerts_aux";
-    let _ =
-        testing::drop_alert_collections(&test_col_name, &test_cutout_col_name, &test_aux_col_name)
-            .await;
-
-    let mut alert_worker = ZtfAlertWorker::new(CONFIG_FILE).await.unwrap();
-    let file_name = "tests/data/alerts/ztf/2695378462115010012.avro";
-    let bytes_content = std::fs::read(file_name).unwrap();
-    let result = alert_worker.process_alert(&bytes_content).await;
-    assert!(result.is_ok(), "Error processing alert: {:?}", result);
-
-    let candids = vec![result.unwrap()];
-
-    info!("received {} candids from redis", candids.len());
-    let out_candids = process_alerts(candids, filter.pipeline, &alert_collection).await;
-    match out_candids {
-        Ok(out) => {
-            assert_eq!(out.len(), 1);
-        }
-        Err(e) => {
-            panic!("Got error: {}", e);
-        }
-    }
-}
-
-#[tokio::test]
-async fn test_filter_no_alerts() {
-    testing::insert_test_filter().await.unwrap();
-    let config_file = conf::load_config("tests/config.test.yaml").unwrap();
-    let db = conf::build_db(&config_file).await.unwrap();
-    let alert_collection = db.collection("ZTF_alerts");
-    let filter_collection = db.collection("filters");
-    let filter_result = ZtfFilter::build(-1, &filter_collection).await;
-    testing::remove_test_filter().await.unwrap();
-    assert!(filter_result.is_ok());
-    let filter = filter_result.unwrap();
-
-    let candids: Vec<i64> = vec![1];
-
-    let out_candids = process_alerts(candids, filter.pipeline, &alert_collection).await;
-    match out_candids {
-        Err(e) => {
-            error!("Error running filter: {}", e);
-        }
-        _ => {}
-    }
-}
diff --git a/tests/test_lsst.rs b/tests/test_lsst.rs
index 2df3a2b4..ebed14f1 100644
--- a/tests/test_lsst.rs
+++ b/tests/test_lsst.rs
@@ -1,7 +1,11 @@
 use boom::{
     alert::{AlertWorker, LsstAlertWorker},
     conf,
-    utils::testing::{drop_alert_from_collections, AlertRandomizerTrait, LsstAlertRandomizer},
+    filter::{FilterWorker, LsstFilterWorker},
+    utils::testing::{
+        drop_alert_from_collections, insert_test_lsst_filter, remove_test_lsst_filter,
+        AlertRandomizerTrait, LsstAlertRandomizer,
+    },
 };
 use mongodb::bson::doc;
 
@@ -97,6 +101,7 @@ async fn test_process_lsst_alert() {
         .find_one(filter.clone())
         .await
         .unwrap();
+
     assert!(alert.is_some());
     let alert = alert.unwrap();
     assert_eq!(alert.get_i64("_id").unwrap(), candid);
@@ -143,3 +148,33 @@ async fn test_process_lsst_alert() {
 
     drop_alert_from_collections(candid, "LSST").await.unwrap();
 }
+
+#[tokio::test]
+async fn test_filter_lsst_alert() {
+    let mut alert_worker = LsstAlertWorker::new(CONFIG_FILE).await.unwrap();
+
+    let (candid, object_id, _ra, _dec, bytes_content) = LsstAlertRandomizer::default().get().await;
+    let result = alert_worker.process_alert(&bytes_content).await.unwrap();
+    assert_eq!(result, candid);
+
+    let filter_id = insert_test_lsst_filter().await.unwrap();
+
+    let mut filter_worker = LsstFilterWorker::new(CONFIG_FILE).await.unwrap();
+    let result = filter_worker.process_alerts(&[format!("{}", candid)]).await;
+
+    assert!(result.is_ok());
+    let alerts_output = result.unwrap();
+    assert_eq!(alerts_output.len(), 1);
+    let alert = &alerts_output[0];
+    assert_eq!(alert.candid, candid);
+    assert_eq!(alert.object_id, format!("{}", object_id));
+    assert_eq!(alert.photometry.len(), 3); // prv_candidates + prv_nondetections
+    let filter_passed = alert
+        .filters
+        .iter()
+        .find(|f| f.filter_id == filter_id)
+        .unwrap();
+    assert_eq!(filter_passed.annotations, "{\"mag_now\":23.15}");
+
+    remove_test_lsst_filter(filter_id).await.unwrap();
+}
diff --git a/tests/test_ztf.rs b/tests/test_ztf.rs
index 5cc86286..1239d516 100644
--- a/tests/test_ztf.rs
+++ b/tests/test_ztf.rs
@@ -1,9 +1,13 @@
 use boom::{
     alert::{AlertWorker, ZtfAlertWorker},
     conf,
+    filter::{FilterWorker, ZtfFilterWorker},
     utils::{
         db::mongify,
-        testing::{drop_alert_from_collections, AlertRandomizerTrait, ZtfAlertRandomizer},
+        testing::{
+            drop_alert_from_collections, insert_test_ztf_filter, remove_test_ztf_filter,
+            AlertRandomizerTrait, ZtfAlertRandomizer,
+        },
     },
 };
 use mongodb::bson::doc;
@@ -220,3 +224,36 @@ async fn test_process_ztf_alert() {
 
     drop_alert_from_collections(candid, "ZTF").await.unwrap();
 }
+
+#[tokio::test]
+async fn test_filter_ztf_alert() {
+    let mut alert_worker = ZtfAlertWorker::new(CONFIG_FILE).await.unwrap();
+
+    let (candid, object_id, _ra, _dec, bytes_content) = ZtfAlertRandomizer::default().get().await;
+    let result = alert_worker.process_alert(&bytes_content).await;
+    assert!(result.is_ok());
+    assert_eq!(result.unwrap(), candid);
+
+    let filter_id = insert_test_ztf_filter().await.unwrap();
+
+    let mut filter_worker = ZtfFilterWorker::new(CONFIG_FILE).await.unwrap();
+    let result = filter_worker
+        .process_alerts(&[format!("1,{}", candid)])
+        .await;
+
+    assert!(result.is_ok());
+    let alerts_output = result.unwrap();
+    assert_eq!(alerts_output.len(), 1);
+    let alert = &alerts_output[0];
+    assert_eq!(alert.candid, candid);
+    assert_eq!(alert.object_id, object_id);
+    assert_eq!(alert.photometry.len(), 11); // prv_candidates + prv_nondetections
+    let filter_passed = alert
+        .filters
+        .iter()
+        .find(|f| f.filter_id == filter_id)
+        .unwrap();
+    assert_eq!(filter_passed.annotations, "{\"mag_now\":14.91}");
+
+    remove_test_ztf_filter(filter_id).await.unwrap();
+}


Reviews:
user: login: Theodlz
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-08T22:06:44Z
user: login: copilot-pull-request-reviewer[bot]
body: 

Copilot reviewed 18 out of 20 changed files in this pull request and generated 1 comment.

<details>
<summary>Files not reviewed (2)</summary>

* **data/filters/lsst.json**: Language not supported
* **tests/data/alerts/ztf/3005140370015010009.avro**: Language not supported
</details>

<details>
<summary>Comments suppressed due to low confidence (1)</summary>

**src/filter/ztf.rs:196**
* The field 'filters_by_permission' is declared in ZtfFilterWorker but is never initialized. Ensure you build this mapping (e.g., by iterating over filters and grouping by permission) before returning the worker.
```
Ok(ZtfFilterWorker {
```
</details>


state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-09T00:21:31Z
user: login: jesaerys
body: Looks good. Just a couple side comments on my part, nothing to keep this from merging.
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-04-12T01:11:19Z
user: login: Theodlz
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-12T01:13:18Z
user: login: Theodlz
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-12T01:13:34Z
user: login: jesaerys
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-12T02:24:10Z
user: login: Theodlz
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-14T06:14:10Z
user: login: Theodlz
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-14T06:14:27Z
user: login: jesaerys
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-14T17:22:06Z
user: login: Theodlz
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-14T17:39:53Z
user: login: jesaerys
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-14T17:51:40Z
user: login: Theodlz
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-14T18:32:54Z

Review comments:
path: tests/data/alerts/ztf/3005140370015010009.avro
user: login: Theodlz
body: I added git lfs in the ML PR for the models, so I'll remove these and will re-add with git lfs once that's on main
created_at: 2025-04-08T22:06:44Z
author_association: COLLABORATOR
diff_hunk: @@ -31,6 +88,129 @@ pub enum FilterError {
     InvalidFilterId,
 }
 
+pub fn parse_programid_candid_tuple(tuple_str: &str) -> Option<(i32, i64)> {
+    // We know that we have the programid first, followed by a comma, and then the candid.
+    // the programid is always a single digit (0-9) and the candid is a larger number.
+    // so we don't know to look for the comma to split the string.
+    // and can directly use the indexes to read the values.
+    // while this makes it very specific to this format, it is twice as fast.
path: src/filter/base.rs
user: login: Copilot
body: The function 'parse_programid_candid_tuple' uses fixed slicing indices on the input string without checking its length, which may panic if the format is unexpected. Consider adding length checks or more robust parsing to safely extract the programid and candid.
```suggestion
    // Validate the input format: must be at least 3 characters long and contain a comma at index 1.
    if tuple_str.len() < 3 || !tuple_str.chars().nth(1).map_or(false, |c| c == ',') {
        return None;
    }

    // Safely slice the string based on the validated format.
```
created_at: 2025-04-09T00:21:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -165,27 +345,121 @@ pub enum FilterWorkerError {
     GetQueueNameError,
     #[error("failed to get filter by queue")]
     GetFilterByQueueError,
+    #[error("failed to send filter results to kafka")]
+    SendKafkaMessageError(#[source] rdkafka::error::KafkaError),
+    #[error("could not find alert by candid")]
+    GetAlertByCandidError(#[source] mongodb::error::Error),
+    #[error("could not find alert")]
+    AlertNotFound,
+    #[error("could not load alert schema")]
+    LoadAlertSchemaError(#[source] apache_avro::Error),
+    #[error("failed to serialize alert to Avro")]
+    SerializeAlertError(#[source] apache_avro::Error),
+    #[error("could not find document field")]
+    MissingFieldError(#[from] mongodb::bson::document::ValueAccessError),
+    #[error("failed to create kafka producer")]
+    KafkaProducerError(#[source] rdkafka::error::KafkaError),
path: src/filter/base.rs
user: login: jesaerys
body: I know I proposed this pattern, but I wonder if this is going to be sustainable. An alternative could be to have one variant for each service we depend on, and only keep specialized variants that are actually needed for control flow.
created_at: 2025-04-12T00:35:21Z
author_association: COLLABORATOR
diff_hunk: @@ -219,109 +191,283 @@ impl FilterWorker for ZtfFilterWorker {
             }
         }
 
-        // create a list of output queues, one for each filter
-        let mut filter_results_queues: HashMap<i32, String> = HashMap::new();
-        for filter in &filters {
-            let queue_name = format!("ZTF_alerts_filter_{}_results_queue", filter.id);
-            filter_results_queues.insert(filter.id, queue_name);
-        }
+        Ok(ZtfFilterWorker {
+            alert_collection,
+            input_queue,
+            output_topic,
+            filters,
+            filters_by_permission,
+        })
+    }
 
-        // in a never ending loop, loop over the queues
-        let client_redis = redis::Client::open("redis://localhost:6379".to_string())
-            .map_err(FilterWorkerError::ConnectRedisError)?;
-        let mut con = client_redis
-            .get_multiplexed_async_connection()
-            .await
-            .map_err(FilterWorkerError::ConnectRedisError)?;
+    fn input_queue_name(&self) -> String {
+        self.input_queue.clone()
+    }
 
-        let command_interval: i64 = 500;
-        let mut command_check_countdown = command_interval;
+    fn output_topic_name(&self) -> String {
+        self.output_topic.clone()
+    }
 
-        loop {
-            if command_check_countdown == 0 {
-                match self.receiver.try_recv() {
-                    Ok(WorkerCmd::TERM) => {
-                        info!("filterworker {} received termination command", self.id);
-                        break;
-                    }
-                    Err(TryRecvError::Disconnected) => {
-                        warn!(
-                            "filter worker {} receiver disconnected, terminating",
-                            self.id
-                        );
-                        break;
-                    }
-                    Err(TryRecvError::Empty) => {
-                        command_check_countdown = command_interval;
+    fn has_filters(&self) -> bool {
+        !self.filters.is_empty()
+    }
+
+    async fn build_alert(
+        &self,
+        candid: i64,
+        filter_results: Vec<FilterResults>,
+    ) -> Result<Alert, FilterWorkerError> {
+        let pipeline = vec![
path: src/filter/ztf.rs
user: login: jesaerys
body: Other than `ZTF_alerts_aux` and `ZTF_alerts_cutouts`, this looks exactly like the LSST one. I wonder if it makes sense to somehow reduce duplication, or if these just look similar by coincidence.
created_at: 2025-04-12T00:55:33Z
author_association: COLLABORATOR
diff_hunk: @@ -219,109 +191,283 @@ impl FilterWorker for ZtfFilterWorker {
             }
         }
 
-        // create a list of output queues, one for each filter
-        let mut filter_results_queues: HashMap<i32, String> = HashMap::new();
-        for filter in &filters {
-            let queue_name = format!("ZTF_alerts_filter_{}_results_queue", filter.id);
-            filter_results_queues.insert(filter.id, queue_name);
-        }
+        Ok(ZtfFilterWorker {
+            alert_collection,
+            input_queue,
+            output_topic,
+            filters,
+            filters_by_permission,
+        })
+    }
 
-        // in a never ending loop, loop over the queues
-        let client_redis = redis::Client::open("redis://localhost:6379".to_string())
-            .map_err(FilterWorkerError::ConnectRedisError)?;
-        let mut con = client_redis
-            .get_multiplexed_async_connection()
-            .await
-            .map_err(FilterWorkerError::ConnectRedisError)?;
+    fn input_queue_name(&self) -> String {
+        self.input_queue.clone()
+    }
 
-        let command_interval: i64 = 500;
-        let mut command_check_countdown = command_interval;
+    fn output_topic_name(&self) -> String {
+        self.output_topic.clone()
+    }
 
-        loop {
-            if command_check_countdown == 0 {
-                match self.receiver.try_recv() {
-                    Ok(WorkerCmd::TERM) => {
-                        info!("filterworker {} received termination command", self.id);
-                        break;
-                    }
-                    Err(TryRecvError::Disconnected) => {
-                        warn!(
-                            "filter worker {} receiver disconnected, terminating",
-                            self.id
-                        );
-                        break;
-                    }
-                    Err(TryRecvError::Empty) => {
-                        command_check_countdown = command_interval;
+    fn has_filters(&self) -> bool {
+        !self.filters.is_empty()
+    }
+
+    async fn build_alert(
+        &self,
+        candid: i64,
+        filter_results: Vec<FilterResults>,
+    ) -> Result<Alert, FilterWorkerError> {
+        let pipeline = vec![
path: src/filter/ztf.rs
user: login: Theodlz
body: They look similar thanks to our efforts to avoid difference between both schemas. But I suspect they might differ as we implement more things. So I'd like to keep these as 2 separate methods.
created_at: 2025-04-12T01:13:18Z
author_association: COLLABORATOR
diff_hunk: @@ -165,27 +345,121 @@ pub enum FilterWorkerError {
     GetQueueNameError,
     #[error("failed to get filter by queue")]
     GetFilterByQueueError,
+    #[error("failed to send filter results to kafka")]
+    SendKafkaMessageError(#[source] rdkafka::error::KafkaError),
+    #[error("could not find alert by candid")]
+    GetAlertByCandidError(#[source] mongodb::error::Error),
+    #[error("could not find alert")]
+    AlertNotFound,
+    #[error("could not load alert schema")]
+    LoadAlertSchemaError(#[source] apache_avro::Error),
+    #[error("failed to serialize alert to Avro")]
+    SerializeAlertError(#[source] apache_avro::Error),
+    #[error("could not find document field")]
+    MissingFieldError(#[from] mongodb::bson::document::ValueAccessError),
+    #[error("failed to create kafka producer")]
+    KafkaProducerError(#[source] rdkafka::error::KafkaError),
path: src/filter/base.rs
user: login: Theodlz
body: Could you provide an example?
created_at: 2025-04-12T01:13:34Z
author_association: COLLABORATOR
diff_hunk: @@ -165,27 +345,121 @@ pub enum FilterWorkerError {
     GetQueueNameError,
     #[error("failed to get filter by queue")]
     GetFilterByQueueError,
+    #[error("failed to send filter results to kafka")]
+    SendKafkaMessageError(#[source] rdkafka::error::KafkaError),
+    #[error("could not find alert by candid")]
+    GetAlertByCandidError(#[source] mongodb::error::Error),
+    #[error("could not find alert")]
+    AlertNotFound,
+    #[error("could not load alert schema")]
+    LoadAlertSchemaError(#[source] apache_avro::Error),
+    #[error("failed to serialize alert to Avro")]
+    SerializeAlertError(#[source] apache_avro::Error),
+    #[error("could not find document field")]
+    MissingFieldError(#[from] mongodb::bson::document::ValueAccessError),
+    #[error("failed to create kafka producer")]
+    KafkaProducerError(#[source] rdkafka::error::KafkaError),
path: src/filter/base.rs
user: login: jesaerys
body: Here's a sketch:

```rust
#[derive(thiserror::Error, Debug)]
pub enum FilterWorkerError {
    ...(specific error variants we need)...
    #[error("error from redis")]
    RedisError(#[from redis::RedisError),
    #[error("error from mongodb")]
    MongodbError(#[from] mongodb::error::Error),
    #[error("error kafka")]
    KafkaError(#[from] rdkafka::error::KafkaError),
    #[error("error from avro")]
    AvroError(#[from] apache_avro::Error),
}
```

So basically, we have a small number of catch-all variants for when the specific root cause of an error isn't important for our program. In fact, we could push this idea further with `anyhow`:

```rust
#[derive(thiserror::Error, Debug)]
pub enum FilterWorkerError {
    ...(specific error variants we need)...
    #[error("oh nos")]
    Error(#[from anyhow::Error),
}
```

As for `specific error variants we need`, those would be anything that we actually need for control flow in boom. I'm having trouble finding an example involving our own error types (maybe that's telling), but for instance we use `TryRecvError` to decide whether to continue looping or issue a warning.

Another possible use case for detailed variants—and kind of what I had in mind in the beginning—was to help track down the specific source of bugs. I've worked on other projects where that's been really useful ("Yes thank you I can see it's a postgres error, but which one, and where is it coming from?!?!?") but for some reason it hasn't really come up for me in boom.
created_at: 2025-04-12T02:24:10Z
author_association: COLLABORATOR
diff_hunk: @@ -165,27 +345,121 @@ pub enum FilterWorkerError {
     GetQueueNameError,
     #[error("failed to get filter by queue")]
     GetFilterByQueueError,
+    #[error("failed to send filter results to kafka")]
+    SendKafkaMessageError(#[source] rdkafka::error::KafkaError),
+    #[error("could not find alert by candid")]
+    GetAlertByCandidError(#[source] mongodb::error::Error),
+    #[error("could not find alert")]
+    AlertNotFound,
+    #[error("could not load alert schema")]
+    LoadAlertSchemaError(#[source] apache_avro::Error),
+    #[error("failed to serialize alert to Avro")]
+    SerializeAlertError(#[source] apache_avro::Error),
+    #[error("could not find document field")]
+    MissingFieldError(#[from] mongodb::bson::document::ValueAccessError),
+    #[error("failed to create kafka producer")]
+    KafkaProducerError(#[source] rdkafka::error::KafkaError),
path: src/filter/base.rs
user: login: Theodlz
body: So in general we want to log errors but keep running our workers. That said, especially early in the project, I'd rather have things fail when we run into unexpected errors so we can handle those and not just ignore them and leave unresolved errors when we move to production.

One thing I actually really dislike about our existing error handling (and that would be even more true if we use a lot more generic errors and less specific error variants) is that we catch errors + log them + continue but we don't log the exact error and even worse do not log WHERE it happened! That makes things really hard to debug. Our error logging should really be more verbose and much easier to track. It's already annoying now, and it will be a real problem later if not solved.

@jesaerys would you mind opening a ticket with your comments (I'll add mine there too) about error handling in general? I think we need to revise/improve how things are done here, to make it easier to develop the app an understand where things fail.
created_at: 2025-04-14T06:14:10Z
author_association: COLLABORATOR
diff_hunk: @@ -165,27 +345,121 @@ pub enum FilterWorkerError {
     GetQueueNameError,
     #[error("failed to get filter by queue")]
     GetFilterByQueueError,
+    #[error("failed to send filter results to kafka")]
+    SendKafkaMessageError(#[source] rdkafka::error::KafkaError),
+    #[error("could not find alert by candid")]
+    GetAlertByCandidError(#[source] mongodb::error::Error),
+    #[error("could not find alert")]
+    AlertNotFound,
+    #[error("could not load alert schema")]
+    LoadAlertSchemaError(#[source] apache_avro::Error),
+    #[error("failed to serialize alert to Avro")]
+    SerializeAlertError(#[source] apache_avro::Error),
+    #[error("could not find document field")]
+    MissingFieldError(#[from] mongodb::bson::document::ValueAccessError),
+    #[error("failed to create kafka producer")]
+    KafkaProducerError(#[source] rdkafka::error::KafkaError),
path: src/filter/base.rs
user: login: Theodlz
body: Please link that error to that conversion/PR for book keeping.
created_at: 2025-04-14T06:14:27Z
author_association: COLLABORATOR
diff_hunk: @@ -165,27 +345,121 @@ pub enum FilterWorkerError {
     GetQueueNameError,
     #[error("failed to get filter by queue")]
     GetFilterByQueueError,
+    #[error("failed to send filter results to kafka")]
+    SendKafkaMessageError(#[source] rdkafka::error::KafkaError),
+    #[error("could not find alert by candid")]
+    GetAlertByCandidError(#[source] mongodb::error::Error),
+    #[error("could not find alert")]
+    AlertNotFound,
+    #[error("could not load alert schema")]
+    LoadAlertSchemaError(#[source] apache_avro::Error),
+    #[error("failed to serialize alert to Avro")]
+    SerializeAlertError(#[source] apache_avro::Error),
+    #[error("could not find document field")]
+    MissingFieldError(#[from] mongodb::bson::document::ValueAccessError),
+    #[error("failed to create kafka producer")]
+    KafkaProducerError(#[source] rdkafka::error::KafkaError),
path: src/filter/base.rs
user: login: jesaerys
body: Yep, absolutely.

Just to be clear, moving toward catch-all variants doesn't mean ignoring errors that we care about. My goal is only to write our error types at the right level of detail, because right now the growing number of variants is starting to seem like more detail than we actually need for control flow. I think our current approach is still good advice in general and it seemed like a good place to start at the time, but as the project grows it just seems like there must be a better way.

It's true that reducing the number of error variants for the sake of developer ergonomics means that operators lose the ability to pinpoint exactly where an error came from by its name and/or error message in the logs. So what do we do?

I think the crux of the issue is that we're relying on the _identify_ of a variant to tell us about the what/why/where of an error in the logs. What we can do instead is get more serious about tracing, in particular, using `span`s with as much automatic instrumentation as possible. Basically, we let tracing tell us all about the context of an error (what/why/where), whatever that error happens to be, even if it's an `anyhow::Error`. This frees us from having to create a new variant for every possible failure mode without sacrificing the quality of our logs.
created_at: 2025-04-14T17:22:06Z
author_association: COLLABORATOR
diff_hunk: @@ -165,27 +345,121 @@ pub enum FilterWorkerError {
     GetQueueNameError,
     #[error("failed to get filter by queue")]
     GetFilterByQueueError,
+    #[error("failed to send filter results to kafka")]
+    SendKafkaMessageError(#[source] rdkafka::error::KafkaError),
+    #[error("could not find alert by candid")]
+    GetAlertByCandidError(#[source] mongodb::error::Error),
+    #[error("could not find alert")]
+    AlertNotFound,
+    #[error("could not load alert schema")]
+    LoadAlertSchemaError(#[source] apache_avro::Error),
+    #[error("failed to serialize alert to Avro")]
+    SerializeAlertError(#[source] apache_avro::Error),
+    #[error("could not find document field")]
+    MissingFieldError(#[from] mongodb::bson::document::ValueAccessError),
+    #[error("failed to create kafka producer")]
+    KafkaProducerError(#[source] rdkafka::error::KafkaError),
path: src/filter/base.rs
user: login: Theodlz
body: Yep I think that as long as we can track `where` and `what` the error was, that's all we need, and that would be better than having a lot of custom variants that don't really tell you where stuff went wrong and why anyways, on top of making things more annoying for the dev.
created_at: 2025-04-14T17:39:53Z
author_association: COLLABORATOR
diff_hunk: @@ -165,27 +345,121 @@ pub enum FilterWorkerError {
     GetQueueNameError,
     #[error("failed to get filter by queue")]
     GetFilterByQueueError,
+    #[error("failed to send filter results to kafka")]
+    SendKafkaMessageError(#[source] rdkafka::error::KafkaError),
+    #[error("could not find alert by candid")]
+    GetAlertByCandidError(#[source] mongodb::error::Error),
+    #[error("could not find alert")]
+    AlertNotFound,
+    #[error("could not load alert schema")]
+    LoadAlertSchemaError(#[source] apache_avro::Error),
+    #[error("failed to serialize alert to Avro")]
+    SerializeAlertError(#[source] apache_avro::Error),
+    #[error("could not find document field")]
+    MissingFieldError(#[from] mongodb::bson::document::ValueAccessError),
+    #[error("failed to create kafka producer")]
+    KafkaProducerError(#[source] rdkafka::error::KafkaError),
path: src/filter/base.rs
user: login: jesaerys
body: I created an issue for this: #105 
created_at: 2025-04-14T17:51:40Z
author_association: COLLABORATOR
diff_hunk: @@ -165,27 +345,121 @@ pub enum FilterWorkerError {
     GetQueueNameError,
     #[error("failed to get filter by queue")]
     GetFilterByQueueError,
+    #[error("failed to send filter results to kafka")]
+    SendKafkaMessageError(#[source] rdkafka::error::KafkaError),
+    #[error("could not find alert by candid")]
+    GetAlertByCandidError(#[source] mongodb::error::Error),
+    #[error("could not find alert")]
+    AlertNotFound,
+    #[error("could not load alert schema")]
+    LoadAlertSchemaError(#[source] apache_avro::Error),
+    #[error("failed to serialize alert to Avro")]
+    SerializeAlertError(#[source] apache_avro::Error),
+    #[error("could not find document field")]
+    MissingFieldError(#[from] mongodb::bson::document::ValueAccessError),
+    #[error("failed to create kafka producer")]
+    KafkaProducerError(#[source] rdkafka::error::KafkaError),
path: src/filter/base.rs
user: login: Theodlz
body: amazing, thanks
created_at: 2025-04-14T18:32:54Z
author_association: COLLABORATOR

PR comments:
user: login: Theodlz
created_at: 2025-04-09T00:22:17Z
body: @jesaerys you'll see that tests and added avro packets aside, the "actual" code added/edited by this PR is half the size displayed by git, and even then a good fraction is just moving existing code around which changes the indentation and github just looses it's mind.
author_association: COLLABORATOR
user: login: Theodlz
created_at: 2025-04-14T18:33:34Z
body: @thanks for the review Jake, I just merged with main and with the new alert randomizer classes this PR is 300 lines shorter which is always nice. I'll go ahead an merge it.
author_association: COLLABORATOR