Repository: RedHatInsights/insights-host-inventory
Pull request #2535: fix(RHINENG-13683): Correctly process multiple messages for the same host in the same MQ batch
Author: login: fstavela
Created: 2025-05-13T13:54:22Z | Merged: 2025-05-22T08:46:02Z

# Overview

This PR is being created to address [RHINENG-13683](https://issues.redhat.com/browse/RHINENG-13683) and [RHINENG-17921](https://issues.redhat.com/browse/RHINENG-17921).

Summary of the issue: When we are processing a payload message, we check if the host already exists in the DB, if yes we update it, if no we create a new host. However, we only flush and commit our changes after a batch of MQ messages have been processed. So, if we are processing 2nd input message for the same host within the same MQ batch, we don't find the existing host (because it is not committed yet) and we create a new one - and we end up with a duplicate.

This PR creates new methods for finding existing hosts by canonical facts in a provided list of hosts. These methods (`multiple_canonical_facts_host_query_in_memory`, `contains_no_incorrect_facts_filter_in_memory`, and `matches_at_least_one_canonical_fact_filter_in_memory`) are creating a python filter queries (NOT sqlalchemy query) which do the same thing as the existing DB query methods, but these filters are intended to be used on an existing in-memory list of hosts, instead of for querying the DB.

Functions like `find_existing_host` are now accepting one additional parameter (`from_hosts`). If this parameter is provided, then they use the new python filter to filter out the provided list of hosts. Otherwise, they use the old DB queries.

In the event loop, we pass the `processed_rows` to the lower-level methods `handle_message` and `process_message`, which use this list to first check if we can find a match inside the `processed_rows`. Only if we can't, then we try to query the DB.

My main concern is around metrics. The `find_existing_host` and other functions are collecting metrics for the dedup process, which will happen only in-memory in this case, so the metrics will be affected by this. Do you think this is an issue?

## PR Checklist

- [ ] Keep PR title short, ideally under 72 characters
- [ ] Descriptive comments provided in complex code blocks
- [ ] Include raw query examples in the PR description, if adding/modifying SQL query
- [ ] Tests: validate optimal/expected output
- [ ] Tests: validate exceptions and failure scenarios
- [ ] Tests: edge cases
- [ ] Recovers or fails gracefully during potential resource outages (e.g. DB, Kafka)
- [ ] Uses [type hinting](https://docs.python.org/3/library/typing.html), if convenient
- [ ] Documentation, if this PR changes the way other services interact with host inventory
- [ ] Links to related PRs

## Secure Coding Practices Documentation Reference

You can find documentation on this checklist [here](https://github.com/RedHatInsights/secure-coding-checklist).

## Secure Coding Checklist

- [ ] Input Validation
- [ ] Output Encoding
- [ ] Authentication and Password Management
- [ ] Session Management
- [ ] Access Control
- [ ] Cryptographic Practices
- [ ] Error Handling and Logging
- [ ] Data Protection
- [ ] Communication Security
- [ ] System Configuration
- [ ] Database Security
- [ ] File Management
- [ ] Memory Management
- [ ] General Coding Practices

## Summary by Sourcery

Refactor host deduplication and message processing to enable correct handling of multiple messages for the same host within a single MQ batch by passing already-processed hosts in memory to lookup and add operations.

Bug Fixes:
- Ensure subsequent messages for the same host in the same MQ batch match the already-processed host instead of creating duplicates.

Enhancements:
- Extend find_existing_host and related lookup methods to accept an optional in-memory host list and branch between DB and in-memory queries.
- Propagate the processed_rows list through handle_message and process_message in MQ consumers and update add_host signature with an existing_host parameter.
- Introduce in-memory filtering functions (multiple_canonical_facts_host_query_in_memory, contains_no_incorrect_facts_filter_in_memory, matches_at_least_one_canonical_fact_filter_in_memory).
- Add type annotations to various functions (parse_operation_message, common_message_parser, update_query_for_owner_id, create_mock_identity_with_org_id, get_staleness_obj, initialize_thread_local_storage, operation_results_to_event_type).
- Add validation in write_add_update_event_message to enforce non-null event_type and platform_metadata.
- Remove unused single_canonical_fact_host_query function.

Tests:
- Update tests to account for the new processed_rows argument in handle_message and process_message signatures.

## Summary by Sourcery

Implement in-memory deduplication of hosts within an MQ batch by extending host lookup functions to accept a list of already-processed hosts and propagating processed rows through message consumers to prevent duplicate host creation.

Bug Fixes:
- Deduplicate multiple messages for the same host in a single MQ batch by matching against already-processed hosts before querying the database

Enhancements:
- Extend find_existing_host and related lookup methods to accept an optional in-memory host list and branch between DB and in-memory queries
- Introduce Python-based in-memory filter helpers for canonical facts matching to mirror existing DB queries
- Propagate processed_rows through HBI message consumers and update event loop and post_process_rows to commit and publish batched events using in-memory deduplication

Tests:
- Add tests for in-batch deduplication, two-host batch processing, and fallback to DB-based deduplication

Chores:
- Add type annotations across multiple functions, validate non-null event payloads, and remove unused single_canonical_fact_host_query function


Diff:
diff --git a/api/filtering/db_filters.py b/api/filtering/db_filters.py
index f0fe42e5e8..f825b0c939 100644
--- a/api/filtering/db_filters.py
+++ b/api/filtering/db_filters.py
@@ -17,6 +17,7 @@
 from api.filtering.db_custom_filters import build_system_profile_filter
 from api.filtering.db_custom_filters import get_host_types_from_filter
 from api.staleness_query import get_staleness_obj
+from app.auth.identity import Identity
 from app.auth.identity import IdentityType
 from app.config import ALL_STALENESS_STATES
 from app.config import HOST_TYPES
@@ -317,13 +318,13 @@ def rbac_permissions_filter(rbac_filter: dict) -> list:
     return _query_filter
 
 
-def update_query_for_owner_id(identity, query):
+def update_query_for_owner_id(identity: Identity, query: Query) -> Query:
     # kafka based requests have dummy identity for working around the identity requirement for CRUD operations
-    logger.debug("identity auth type: %s", identity.auth_type)
-    if identity and identity.identity_type == IdentityType.SYSTEM:
-        return query.filter(and_(Host.system_profile_facts["owner_id"].as_string() == identity.system["cn"]))
-    else:
-        return query
+    if identity:
+        logger.debug("identity auth type: %s", identity.auth_type)
+        if identity.identity_type == IdentityType.SYSTEM:
+            return query.filter(and_(Host.system_profile_facts["owner_id"].as_string() == identity.system["cn"]))
+    return query
 
 
 def query_filters(
diff --git a/api/staleness_query.py b/api/staleness_query.py
index 18e307e10d..c38a03613c 100644
--- a/api/staleness_query.py
+++ b/api/staleness_query.py
@@ -2,13 +2,14 @@
 
 from app.logging import get_logger
 from app.models import Staleness
+from app.staleness_serialization import AttrDict
 from app.staleness_serialization import build_serialized_acc_staleness_obj
 from app.staleness_serialization import build_staleness_sys_default
 
 logger = get_logger(__name__)
 
 
-def get_staleness_obj(org_id):
+def get_staleness_obj(org_id: str) -> AttrDict:
     try:
         staleness = Staleness.query.filter(Staleness.org_id == org_id).one()
         logger.info("Using custom account staleness")
diff --git a/app/__init__.py b/app/__init__.py
index ce1808f150..e6378c2e01 100644
--- a/app/__init__.py
+++ b/app/__init__.py
@@ -230,7 +230,7 @@ def process_system_profile_spec():
         return process_spec(yaml.safe_load(fp)["$defs"]["SystemProfile"]["properties"])
 
 
-def create_app(runtime_environment):
+def create_app(runtime_environment) -> connexion.FlaskApp:
     # This feels like a hack but it is needed.  The logging configuration
     # needs to be setup before the flask app is initialized.
     configure_logging()
diff --git a/app/auth/identity.py b/app/auth/identity.py
index 97b3566cb9..5816426a57 100644
--- a/app/auth/identity.py
+++ b/app/auth/identity.py
@@ -177,7 +177,7 @@ class ServiceAccountIdentitySchema(IdentityBaseSchema):
 
 # Messages from the system_profile topic don't need to provide a real Identity,
 # So this helper function creates a basic User-type identity from the host data.
-def create_mock_identity_with_org_id(org_id):
+def create_mock_identity_with_org_id(org_id: str) -> Identity:
     return Identity(
         {
             "org_id": org_id,
diff --git a/app/queue/events.py b/app/queue/events.py
index f40ca2aa5b..cdc6c4c138 100644
--- a/app/queue/events.py
+++ b/app/queue/events.py
@@ -148,5 +148,5 @@ def build_event(event_type, host, **kwargs):
         return result
 
 
-def operation_results_to_event_type(results):
+def operation_results_to_event_type(results) -> EventType:
     return EventType[results.name]
diff --git a/app/queue/host_mq.py b/app/queue/host_mq.py
index 1916fff82e..4fd2f38215 100644
--- a/app/queue/host_mq.py
+++ b/app/queue/host_mq.py
@@ -5,10 +5,12 @@
 import sys
 from copy import deepcopy
 from functools import partial
+from typing import Any
+from typing import Callable
 from uuid import UUID
 
 from confluent_kafka import Consumer
-from flask.app import Flask
+from connexion import FlaskApp
 from marshmallow import Schema
 from marshmallow import ValidationError
 from marshmallow import fields
@@ -58,6 +60,7 @@
 from app.serialization import remove_null_canonical_facts
 from app.serialization import serialize_group
 from app.serialization import serialize_host
+from app.staleness_serialization import AttrDict
 from lib import group_repository
 from lib import host_repository
 from lib.db import session_guard
@@ -102,7 +105,15 @@ class DebeziumEnvelopeSchema(Schema):
 
 # Helper class to facilitate batch operations
 class OperationResult:
-    def __init__(self, hr, pm, st, so, et, sl):
+    def __init__(
+        self,
+        hr: Host,
+        pm: dict[str, Any] | None,
+        st: Timestamps | None,
+        so: AttrDict | None,
+        et: EventType | None,
+        sl: Callable,
+    ):
         self.host_row = hr
         self.platform_metadata = pm
         self.staleness_timestamps = st
@@ -115,7 +126,7 @@ class HBIMessageConsumerBase:
     def __init__(
         self,
         consumer: Consumer,
-        flask_app: Flask,
+        flask_app: FlaskApp,
         event_producer: EventProducer,
         notification_event_producer: EventProducer,
     ) -> None:
@@ -123,6 +134,7 @@ def __init__(
         self.flask_app = flask_app
         self.event_producer = event_producer
         self.notification_event_producer = notification_event_producer
+        self.processed_rows: list[OperationResult] = []
 
     def process_message(self, *args, **kwargs):
         raise NotImplementedError("Not implemented in the HBIMessageConsumerBase class")
@@ -131,13 +143,13 @@ def process_message(self, *args, **kwargs):
     def handle_message(self, *args, **kwargs) -> OperationResult | None:
         raise NotImplementedError("Not implemented in the HBIMessageConsumerBase class")
 
-    def post_process_rows(self, processed_rows: list[OperationResult]):
+    def post_process_rows(self) -> None:
         pass  # No action is taken by default
 
     def event_loop(self, interrupt):
         with self.flask_app.app.app_context():
             while not interrupt():
-                processed_rows: list[OperationResult] = []
+                self.processed_rows = []
                 with session_guard(db.session), db.session.no_autoflush:
                     messages = self.consumer.consume(
                         num_messages=inventory_config().mq_db_batch_max_messages,
@@ -158,7 +170,7 @@ def event_loop(self, interrupt):
                             logger.debug("Message received")
 
                             try:
-                                processed_rows.append(self.handle_message(msg.value()))
+                                self.processed_rows.append(self.handle_message(msg.value()))
                                 metrics.consumed_message_size.observe(len(str(msg).encode("utf-8")))
                                 metrics.ingress_message_handler_success.inc()
                             except OperationalError as oe:
@@ -172,12 +184,12 @@ def event_loop(self, interrupt):
                                 metrics.ingress_message_handler_failure.inc()
                                 logger.exception("Unable to process message", extra={"incoming_message": msg.value()})
 
-                    self.post_process_rows(processed_rows)
+                    self.post_process_rows()
 
 
 class WorkspaceMessageConsumer(HBIMessageConsumerBase):
     @metrics.ingress_message_handler_time.time()
-    def handle_message(self, message):
+    def handle_message(self, message: str | bytes):
         payload_schema = parse_operation_message(message, DebeziumEnvelopeSchema)
         validated_operation_msg = parse_operation_message(payload_schema["payload"], WorkspaceOperationSchema)
         initialize_thread_local_storage(None)  # No request_id for workspace MQ
@@ -226,7 +238,7 @@ def handle_message(self, message):
 
 class HostMessageConsumer(HBIMessageConsumerBase):
     @metrics.ingress_message_handler_time.time()
-    def handle_message(self, message) -> OperationResult:
+    def handle_message(self, message: str | bytes) -> OperationResult:
         validated_operation_msg = parse_operation_message(message, HostOperationSchema)
         platform_metadata = validated_operation_msg.get("platform_metadata", {})
 
@@ -277,25 +289,37 @@ def handle_message(self, message) -> OperationResult:
                 )
                 raise
 
-    def post_process_rows(self, processed_rows: list[OperationResult]) -> None:
+    def post_process_rows(self) -> None:
         try:
-            if len(processed_rows) > 0:
+            if len(self.processed_rows) > 0:
                 db.session.commit()
                 # The above session is automatically committed or rolled back.
                 # Now we need to send out messages for the batch of hosts we just processed.
-                write_message_batch(self.event_producer, self.notification_event_producer, processed_rows)
+                write_message_batch(self.event_producer, self.notification_event_producer, self.processed_rows)
 
         except StaleDataError as exc:
-            metrics.ingress_message_handler_failure.inc(amount=len(processed_rows))
+            metrics.ingress_message_handler_failure.inc(amount=len(self.processed_rows))
             logger.error(
-                f"Session data is stale; failed to commit data from {len(processed_rows)} payloads.",
+                f"Session data is stale; failed to commit data from {len(self.processed_rows)} payloads.",
                 exc_info=exc,
             )
             db.session.rollback()
 
 
 class IngressMessageConsumer(HostMessageConsumer):
-    def process_message(self, host_data, platform_metadata, operation_args=None):
+    def process_message(
+        self,
+        host_data: dict[str, Any],
+        platform_metadata: dict[str, Any],
+        operation_args: dict[str, Any] | None = None,
+    ) -> tuple[Host, host_repository.AddHostResult, Identity, Callable]:
+        """
+        Returned values from this method are:
+        - host_row: the host that was added or updated
+        - add_result: the result of the add operation (created or updated)
+        - identity: the currently used identity
+        - success_logger: a callable that takes a host and logs the current MQ operation
+        """
         if operation_args is None:
             operation_args = {}
 
@@ -309,7 +333,10 @@ def process_message(self, host_data, platform_metadata, operation_args=None):
                 input_host = _set_owner(input_host, identity)
 
             log_add_host_attempt(logger, input_host, sp_fields_to_log, identity)
-            host_row, add_result = host_repository.add_host(input_host, identity, operation_args=operation_args)
+            processed_hosts = [result.host_row for result in self.processed_rows]
+            host_row, add_result = host_repository.add_host(
+                input_host, identity, operation_args=operation_args, existing_hosts=processed_hosts
+            )
 
             # If this is a new host, assign it to the "ungrouped hosts" group/workspace
             if add_result == host_repository.AddHostResult.created and get_flag_value(
@@ -342,7 +369,19 @@ def process_message(self, host_data, platform_metadata, operation_args=None):
 
 
 class SystemProfileMessageConsumer(HostMessageConsumer):
-    def process_message(self, host_data, platform_metadata, operation_args=None):  # noqa: ARG002, required by process_message
+    def process_message(
+        self,
+        host_data: dict[str, Any],
+        platform_metadata: dict[str, Any],  # noqa: ARG002, required by process_message
+        operation_args: dict[str, Any] | None = None,
+    ) -> tuple[Host, host_repository.AddHostResult, Identity, Callable]:
+        """
+        Returned values from this method are:
+        - output_host: the host that was updated
+        - update_result: the result of the 'update_system_profile' operation (AddHostResult.updated)
+        - identity: the currently used identity
+        - success_logger: a callable that takes a host and logs the current MQ operation
+        """
         if operation_args is None:
             operation_args = {}
 
@@ -467,7 +506,7 @@ def _validate_json_object_for_utf8(json_object):
 
 
 @metrics.ingress_message_parsing_time.time()
-def parse_operation_message(message, schema: Schema):
+def parse_operation_message(message: str | bytes, schema: type[Schema]):
     parsed_message = common_message_parser(message)
 
     try:
@@ -545,6 +584,11 @@ def write_delete_event_message(event_producer: EventProducer, result: OperationR
 def write_add_update_event_message(
     event_producer: EventProducer, notification_event_producer: EventProducer, result: OperationResult
 ):
+    if result.event_type is None or result.platform_metadata is None:
+        # This should never happen, but OperationResult allows None values for these fields.
+        logger.error("Invalid operation result. Event type and platform metadata must be provided.")
+        raise ValueError("Event type and platform metadata must be provided.")
+
     # The request ID in the headers is fetched from threadctx.request_id
     request_id = result.platform_metadata.get("request_id")
     initialize_thread_local_storage(request_id, result.host_row.org_id, result.host_row.account)
@@ -610,7 +654,7 @@ def write_message_batch(
                 logger.exception("Error while producing message", exc_info=exc)
 
 
-def initialize_thread_local_storage(request_id: str, org_id: str | None = None, account: str | None = None):
+def initialize_thread_local_storage(request_id: str | None, org_id: str | None = None, account: str | None = None):
     threadctx.request_id = request_id
     if org_id:
         threadctx.org_id = org_id
diff --git a/app/queue/mq_common.py b/app/queue/mq_common.py
index b4fa2042fd..bfd5c04d4b 100644
--- a/app/queue/mq_common.py
+++ b/app/queue/mq_common.py
@@ -1,3 +1,5 @@
+from __future__ import annotations
+
 import json
 
 from app.logging import get_logger
@@ -7,7 +9,7 @@
 
 
 @metrics.common_message_parsing_time.time()
-def common_message_parser(message):
+def common_message_parser(message: str | bytes):
     try:
         # Due to RHCLOUD-3610 we're receiving messages with invalid unicode code points (invalid surrogate pairs)
         # Python pretty much ignores that but it is not possible to store such strings in the database (db INSERTS
diff --git a/lib/host_repository.py b/lib/host_repository.py
index be583d42cf..77da79ec40 100644
--- a/lib/host_repository.py
+++ b/lib/host_repository.py
@@ -1,12 +1,17 @@
 from __future__ import annotations
 
 from enum import Enum
+from typing import Any
+from typing import Callable
 from uuid import UUID
 
 from flask import current_app
+from flask_sqlalchemy.query import Query
 from sqlalchemy import and_
 from sqlalchemy import not_
 from sqlalchemy import or_
+from sqlalchemy.sql.elements import BinaryExpression
+from sqlalchemy.sql.elements import BooleanClauseList
 
 from api.filtering.db_filters import find_stale_host_in_window
 from api.filtering.db_filters import stale_timestamp_filter
@@ -21,6 +26,7 @@
 from app.models import Group
 from app.models import Host
 from app.models import HostGroupAssoc
+from app.models import LimitedHost
 from app.serialization import serialize_staleness_to_dict
 from app.staleness_serialization import get_sys_default_staleness
 from lib import metrics
@@ -28,8 +34,8 @@
 from lib.feature_flags import get_flag_value
 
 __all__ = (
+    "AddHostResult",
     "add_host",
-    "single_canonical_fact_host_query",
     "multiple_canonical_facts_host_query",
     "create_new_host",
     "find_existing_host",
@@ -59,7 +65,11 @@
 
 
 def add_host(
-    input_host: Host, identity: Identity, update_system_profile: bool = True, operation_args: dict | None = None
+    input_host: Host,
+    identity: Identity,
+    update_system_profile: bool = True,
+    operation_args: dict | None = None,
+    existing_hosts: list[Host] | None = None,
 ) -> tuple[Host, AddHostResult]:
     """
     Add or update a host
@@ -67,32 +77,44 @@ def add_host(
     Required parameters:
      - at least one of the canonical facts fields is required
      - org_id
+
+     The only supported argument in the operation_args for now is "defer_to_reporter".
+     It is documented here:
+     https://inscope.corp.redhat.com/docs/default/Component/consoledot-pages/services/inventory/#expected-message-format
     """
     if operation_args is None:
         operation_args = {}
-    existing_host = find_existing_host(identity, input_host.canonical_facts)
-    if existing_host:
+
+    matched_host = None
+    if existing_hosts:
+        # First, try to match the host in memory - from the provided list of existing hosts
+        matched_host = find_existing_host(identity, input_host.canonical_facts, existing_hosts)
+    if matched_host is None:
+        # If the list of existing hosts was not provided, or the match was not found, try querying DB
+        matched_host = find_existing_host(identity, input_host.canonical_facts)
+
+    if matched_host:
         defer_to_reporter = operation_args.get("defer_to_reporter", None)
         if defer_to_reporter is not None:
             logger.debug("host_repository.add_host: defer_to_reporter = %s", defer_to_reporter)
-            if not existing_host.reporter_stale(defer_to_reporter):
+            if not matched_host.reporter_stale(defer_to_reporter):
                 logger.debug("host_repository.add_host: setting update_system_profile = False")
                 update_system_profile = False
 
-        return update_existing_host(existing_host, input_host, update_system_profile)
+        return update_existing_host(matched_host, input_host, update_system_profile)
     else:
         return create_new_host(input_host)
 
 
 @metrics.host_dedup_processing_time.time()
-def find_existing_host(identity: Identity, canonical_facts: dict) -> Host | None:
-    logger.debug("find_existing_host(%s, %s)", identity, canonical_facts)
-    existing_host = _find_host_by_elevated_ids(identity, canonical_facts)
+def find_existing_host(identity: Identity, canonical_facts: dict, from_hosts: list[Host] | None = None) -> Host | None:
+    logger.debug(f"find_existing_host({identity}, {canonical_facts}, {from_hosts})")
+    existing_host = _find_host_by_elevated_ids(identity, canonical_facts, from_hosts)
 
     if existing_host or current_app.config["USE_SUBMAN_ID"]:
         return existing_host
 
-    existing_host = find_host_by_multiple_canonical_facts(identity, canonical_facts)
+    existing_host = find_host_by_multiple_canonical_facts(identity, canonical_facts, from_hosts)
 
     return existing_host
 
@@ -103,8 +125,28 @@ def find_existing_host_by_id(identity: Identity, host_id: str) -> Host | None:
     return find_non_culled_hosts(query, identity).order_by(Host.modified_on.desc()).first()
 
 
+def _find_host_by_multiple_facts_in_db_or_in_memory(
+    identity: Identity, canonical_facts: dict, from_hosts: list[Host] | None = None
+) -> Host | None:
+    if from_hosts:
+        with metrics.find_host_by_facts_in_memory.time():
+            existing_hosts = list(
+                filter(multiple_canonical_facts_host_query_in_memory(identity, canonical_facts), from_hosts)
+            )
+        return existing_hosts[0] if existing_hosts else None
+
+    with metrics.find_host_by_facts_in_db.time():
+        return (
+            multiple_canonical_facts_host_query(identity, canonical_facts, restrict_to_owner_id=False)
+            .order_by(Host.modified_on.desc())
+            .first()
+        )
+
+
 @metrics.find_host_using_elevated_ids.time()
-def _find_host_by_elevated_ids(identity: Identity, canonical_facts: dict) -> Host | None:
+def _find_host_by_elevated_ids(
+    identity: Identity, canonical_facts: dict, from_hosts: list[Host] | None = None
+) -> Host | None:
     elevated_facts = {}
     elevated_keys = []
     immutable_facts = {}
@@ -133,11 +175,7 @@ def _find_host_by_elevated_ids(identity: Identity, canonical_facts: dict) -> Hos
 
     # First search based on immutable elevated canonical facts.
     if immutable_facts:
-        existing_host = (
-            multiple_canonical_facts_host_query(identity, immutable_facts, False)
-            .order_by(Host.modified_on.desc())
-            .first()
-        )
+        existing_host = _find_host_by_multiple_facts_in_db_or_in_memory(identity, immutable_facts, from_hosts)
         if existing_host:
             return existing_host
 
@@ -155,26 +193,14 @@ def _find_host_by_elevated_ids(identity: Identity, canonical_facts: dict) -> Hos
         if compound_fact := COMPOUND_CANONICAL_FACTS_MAP.get(target_key):  # noqa: SIM102
             if compound_fact_val := canonical_facts.get(compound_fact):
                 target_facts[compound_fact] = compound_fact_val
-        existing_host = (
-            multiple_canonical_facts_host_query(identity, target_facts, False)
-            .order_by(Host.modified_on.desc())
-            .first()
-        )
+
+        existing_host = _find_host_by_multiple_facts_in_db_or_in_memory(identity, target_facts, from_hosts)
         if existing_host:
             return existing_host
 
     return None
 
 
-def single_canonical_fact_host_query(identity, canonical_fact, value, restrict_to_owner_id=True):
-    query = Host.query.filter(
-        (Host.org_id == identity.org_id) & (Host.canonical_facts[canonical_fact].astext == value)
-    )
-    if restrict_to_owner_id:
-        query = update_query_for_owner_id(identity, query)
-    return find_non_culled_hosts(query, identity)
-
-
 # The CanonicalFactsSchema ensures that both components of compound
 # canonical facts are provided. This check is included to detect any
 # unforeseen issues.
@@ -184,7 +210,9 @@ def _check_compound_canonical_facts(canonical_facts):
             raise InventoryException(title="Invalid request", detail=f"Unpaired compound fact: {key} missing {value}")
 
 
-def multiple_canonical_facts_host_query(identity, canonical_facts, restrict_to_owner_id=True):
+def multiple_canonical_facts_host_query(
+    identity: Identity, canonical_facts: dict[str, Any], restrict_to_owner_id: bool = True
+) -> Query:
     _check_compound_canonical_facts(canonical_facts)
 
     query = Host.query.filter(
@@ -197,30 +225,40 @@ def multiple_canonical_facts_host_query(identity, canonical_facts, restrict_to_o
     return find_non_culled_hosts(query, identity)
 
 
-def find_host_by_multiple_canonical_facts(identity: Identity, canonical_facts: dict) -> Host | None:
+def multiple_canonical_facts_host_query_in_memory(
+    identity: Identity, canonical_facts: dict[str, Any]
+) -> Callable[[Host], bool]:
+    def _query(host: Host) -> bool:
+        return (
+            host.org_id == identity.org_id
+            and contains_no_incorrect_facts_filter_in_memory(host, canonical_facts)
+            and matches_at_least_one_canonical_fact_filter_in_memory(host, canonical_facts)
+        )
+
+    return _query
+
+
+def find_host_by_multiple_canonical_facts(
+    identity: Identity, canonical_facts: dict, from_hosts: list[Host] | None = None
+) -> Host | None:
     """
     Returns first match for a host containing given canonical facts
     """
-    logger.debug("find_host_by_multiple_canonical_facts(%s)", canonical_facts)
+    logger.debug(f"find_host_by_multiple_canonical_facts({canonical_facts}, {from_hosts})")
 
     # If no canonical facts are supplied, it can't match anything.
     # Just in case the last canonical fact was removed during the process.
     if not canonical_facts:
         return None
 
-    host = (
-        multiple_canonical_facts_host_query(identity, canonical_facts, restrict_to_owner_id=False)
-        .order_by(Host.modified_on.desc())
-        .first()
-    )
-
+    host = _find_host_by_multiple_facts_in_db_or_in_memory(identity, canonical_facts, from_hosts)
     if host:
-        logger.debug("Found existing host using canonical_fact match: %s", host)
+        logger.debug(f"Found existing host using canonical_fact match: {host}")
 
     return host
 
 
-def find_hosts_by_staleness(staleness_types, query, identity):
+def find_hosts_by_staleness(staleness_types: list[str], query: Query, identity: Identity) -> Query:
     logger.debug("find_hosts_by_staleness(%s)", staleness_types)
     staleness_obj = serialize_staleness_to_dict(get_staleness_obj(identity.org_id))
     staleness_conditions = [
@@ -287,7 +325,7 @@ def find_hosts_sys_default_staleness(staleness_types):
     return or_(False, *staleness_conditions)
 
 
-def find_non_culled_hosts(query, identity):
+def find_non_culled_hosts(query: Query, identity: Identity) -> Query:
     return find_hosts_by_staleness(ALL_STALENESS_STATES, query, identity)
 
 
@@ -318,11 +356,11 @@ def update_existing_host(
     return existing_host, AddHostResult.updated
 
 
-def contains_no_incorrect_facts_filter(canonical_facts):
+def contains_no_incorrect_facts_filter(canonical_facts: dict[str, Any]) -> BinaryExpression:
     # Does not contain any incorrect CF values
     # Incorrect value = AND( key exists, NOT( contains key:value ) )
     # -> NOT( OR( *Incorrect values ) )
-    filter_ = ()
+    filter_: tuple = ()
     for key, value in canonical_facts.items():
         filter_ += (
             and_(Host.canonical_facts.has_key(key), not_(Host.canonical_facts.contains({key: value}))),  # noqa: W601
@@ -331,11 +369,19 @@ def contains_no_incorrect_facts_filter(canonical_facts):
     return not_(or_(*filter_))
 
 
-def matches_at_least_one_canonical_fact_filter(canonical_facts):
+def contains_no_incorrect_facts_filter_in_memory(host: Host, canonical_facts: dict[str, Any]) -> bool:
+    def _per_fact_query(host: Host, key: str, value: Any) -> bool:
+        # Either the key is not present in the host, or the fact matches
+        return key not in host.canonical_facts or host.canonical_facts[key] == value
+
+    return all(_per_fact_query(host, key, value) for key, value in canonical_facts.items())
+
+
+def matches_at_least_one_canonical_fact_filter(canonical_facts: dict[str, Any]) -> BooleanClauseList:
     # Contains at least one correct CF value
     # Correct value = contains key:value
     # -> OR( *correct values )
-    filter_ = ()
+    filter_: tuple = ()
     for key, value in canonical_facts.items():
         # Don't include the second component of compound canonical facts
         # in the OR logic of the query. It was already included in the
@@ -347,7 +393,11 @@ def matches_at_least_one_canonical_fact_filter(canonical_facts):
     return or_(*filter_)
 
 
-def update_system_profile(input_host, identity):
+def matches_at_least_one_canonical_fact_filter_in_memory(host: Host, canonical_facts: dict[str, Any]) -> bool:
+    return any(host.canonical_facts.get(key) == value for key, value in canonical_facts.items())
+
+
+def update_system_profile(input_host: Host | LimitedHost, identity: Identity):
     if not input_host.system_profile_facts:
         raise InventoryException(
             title="Invalid request", detail="Cannot update System Profile, since no System Profile data was provided."
diff --git a/lib/metrics.py b/lib/metrics.py
index f5e125aeb0..925263fc56 100644
--- a/lib/metrics.py
+++ b/lib/metrics.py
@@ -8,6 +8,14 @@
     "inventory_find_host_using_elevated_ids_processing_seconds",
     "Time spent looking for existing host using the elevated ids",
 )
+find_host_by_facts_in_db = Summary(
+    "inventory_find_host_by_facts_in_db_processing_seconds",
+    "Time spent looking for existing host by canonical facts in db",
+)
+find_host_by_facts_in_memory = Summary(
+    "inventory_find_host_by_facts_in_memory_processing_seconds",
+    "Time spent looking for existing host by canonical facts in memory (during MQ batch processing)",
+)
 new_host_commit_processing_time = Summary(
     "inventory_new_host_commit_seconds", "Time spent committing a new host to the database"
 )
diff --git a/tests/fixtures/app_fixtures.py b/tests/fixtures/app_fixtures.py
index 5e33d08173..1337a5e51d 100644
--- a/tests/fixtures/app_fixtures.py
+++ b/tests/fixtures/app_fixtures.py
@@ -1,4 +1,7 @@
+from collections.abc import Generator
+
 import pytest
+from connexion import FlaskApp
 from sqlalchemy import text as sa_text
 
 from app import create_app
@@ -9,7 +12,7 @@
 
 
 @pytest.fixture(scope="session")
-def new_flask_app(database):  # noqa: ARG001
+def new_flask_app(database: str) -> Generator[FlaskApp]:  # noqa: ARG001
     application = create_app(RuntimeEnvironment.TEST)
     with application.app.app_context():
         db.session.execute(sa_text("CREATE SCHEMA IF NOT EXISTS hbi"))
@@ -23,13 +26,13 @@ def new_flask_app(database):  # noqa: ARG001
 
 
 @pytest.fixture(scope="function")
-def flask_app(new_flask_app):
+def flask_app(new_flask_app: FlaskApp) -> Generator[FlaskApp]:
     yield new_flask_app
 
     clean_tables()
 
 
 @pytest.fixture(scope="function")
-def inventory_config(flask_app):
+def inventory_config(flask_app: FlaskApp):
     yield flask_app.app.config["INVENTORY_CONFIG"]
     flask_app.app.config["INVENTORY_CONFIG"] = Config(RuntimeEnvironment.TEST)
diff --git a/tests/fixtures/db_fixtures.py b/tests/fixtures/db_fixtures.py
index baba78a8fc..5155dd700a 100644
--- a/tests/fixtures/db_fixtures.py
+++ b/tests/fixtures/db_fixtures.py
@@ -1,6 +1,12 @@
+from __future__ import annotations
+
 import os
+from collections.abc import Generator
+from typing import Any
+from typing import Callable
 
 import pytest
+from connexion import FlaskApp
 from sqlalchemy_utils import create_database
 from sqlalchemy_utils import database_exists
 from sqlalchemy_utils import drop_database
@@ -25,7 +31,7 @@
 
 
 @pytest.fixture(scope="session")
-def database_name():
+def database_name() -> Generator[None]:
     db_data = {
         "INVENTORY_DB_NAME": os.getenv("INVENTORY_DB_NAME", "insights"),
         "INVENTORY_DB_PASS": os.getenv("INVENTORY_DB_PASS", "insights"),
@@ -39,7 +45,7 @@ def database_name():
 
 
 @pytest.fixture(scope="session")
-def database(database_name):  # noqa: ARG001
+def database(database_name: None) -> Generator[str]:  # noqa: ARG001
     config = Config(RuntimeEnvironment.TEST)
     if not database_exists(config.db_uri):
         create_database(config.db_uri)
@@ -73,6 +79,22 @@ def _db_get_host_by_insights_id(insights_id):
     return _db_get_host_by_insights_id
 
 
+@pytest.fixture()
+def db_get_hosts_by_subman_id(flask_app: FlaskApp) -> Callable[[str], list[Host]]:  # noqa: ARG001
+    def _db_get_hosts_by_insights_id(subman_id: str) -> list[Host]:
+        return Host.query.filter(Host.canonical_facts["subscription_manager_id"].astext == subman_id).all()
+
+    return _db_get_hosts_by_insights_id
+
+
+@pytest.fixture()
+def db_get_hosts_by_display_name(flask_app: FlaskApp) -> Callable[[str], list[Host]]:  # noqa: ARG001
+    def _db_get_hosts_by_display_name(display_name: str) -> list[Host]:
+        return Host.query.filter(Host.display_name == display_name).all()
+
+    return _db_get_hosts_by_display_name
+
+
 @pytest.fixture(scope="function")
 def db_get_group_by_id(flask_app):  # noqa: ARG001
     def _db_get_group_by_id(group_id):
@@ -114,8 +136,10 @@ def _db_get_groups_for_host(host_id):
 
 
 @pytest.fixture(scope="function")
-def db_create_host(flask_app):  # noqa: ARG001
-    def _db_create_host(identity=SYSTEM_IDENTITY, host=None, extra_data=None):
+def db_create_host(flask_app: FlaskApp) -> Callable[..., Host]:  # noqa: ARG001
+    def _db_create_host(
+        identity: dict[str, Any] = SYSTEM_IDENTITY, host: Host | None = None, extra_data: dict[str, Any] | None = None
+    ) -> Host:
         extra_data = extra_data or {}
         host = host or minimal_db_host(org_id=identity["org_id"], account=identity["account_number"], **extra_data)
         db.session.add(host)
diff --git a/tests/fixtures/mq_fixtures.py b/tests/fixtures/mq_fixtures.py
index 1a9b595cfd..13f76fa952 100644
--- a/tests/fixtures/mq_fixtures.py
+++ b/tests/fixtures/mq_fixtures.py
@@ -1,9 +1,11 @@
 import json
+from collections.abc import Generator
 from datetime import datetime
 from datetime import timezone
 from unittest.mock import patch
 
 import pytest
+from connexion import FlaskApp
 
 from app.models import db
 from app.queue.event_producer import EventProducer
@@ -153,7 +155,7 @@ def notification_kafka_producer(mocker):
 
 
 @pytest.fixture(scope="function")
-def event_producer(flask_app, kafka_producer):  # noqa: ARG001
+def event_producer(flask_app: FlaskApp, kafka_producer) -> Generator[EventProducer]:  # noqa: ARG001
     config = flask_app.app.config["INVENTORY_CONFIG"]
     flask_app.app.event_producer = EventProducer(config, config.event_topic)
     yield flask_app.app.event_producer
diff --git a/tests/helpers/db_utils.py b/tests/helpers/db_utils.py
index 3b57e57ac4..a008773abf 100644
--- a/tests/helpers/db_utils.py
+++ b/tests/helpers/db_utils.py
@@ -1,6 +1,7 @@
 import logging
 from datetime import timedelta
 from random import randint
+from typing import Any
 
 from sqlalchemy.exc import InvalidRequestError
 
@@ -39,12 +40,12 @@ def _clean_tables():
     _clean_tables()
 
 
-def minimal_db_host(**values):
+def minimal_db_host(**values) -> Host:
     data = minimal_db_host_dict(**values)
     return Host(**data)
 
 
-def minimal_db_host_dict(**values):
+def minimal_db_host_dict(**values) -> dict[str, Any]:
     data = {
         "canonical_facts": {"insights_id": generate_uuid()},
         "stale_timestamp": (now() + timedelta(days=randint(1, 7))),
diff --git a/tests/helpers/test_utils.py b/tests/helpers/test_utils.py
index 54fa4d1c57..66320a7905 100644
--- a/tests/helpers/test_utils.py
+++ b/tests/helpers/test_utils.py
@@ -95,7 +95,7 @@ def set_environment(new_env=None):
     patched_dict.stop()
 
 
-def _base_host_data(**values):
+def _base_host_data(**values) -> dict[str, Any]:
     return {
         "org_id": USER_IDENTITY["org_id"],
         "display_name": "test" + generate_random_string(),
@@ -115,7 +115,7 @@ def base_host(**values):
 # This method returns a host wrapper that contains a single canonical fact, making it
 # a minimal host that's valid for use with mq_create_or_update_host().
 # It should be used in test cases where specific canonical facts are not needed.
-def minimal_host(**values):
+def minimal_host(**values) -> HostWrapper:
     host_wrapper = HostWrapper(_base_host_data(**values))
     host_wrapper.bios_uuid = generate_uuid()
     return host_wrapper
diff --git a/tests/test_host_mq_service.py b/tests/test_host_mq_service.py
index 09fecefc73..540f61fac8 100644
--- a/tests/test_host_mq_service.py
+++ b/tests/test_host_mq_service.py
@@ -1,14 +1,19 @@
+from __future__ import annotations
+
 import json
 import logging
 from copy import deepcopy
 from datetime import datetime
 from datetime import timedelta
 from types import SimpleNamespace
+from typing import Callable
 from unittest import mock
 from unittest.mock import patch
 
 import marshmallow
 import pytest
+from connexion import FlaskApp
+from pytest_mock import MockerFixture
 from sqlalchemy.exc import IntegrityError
 from sqlalchemy.exc import OperationalError
 from sqlalchemy.orm.exc import StaleDataError
@@ -18,6 +23,8 @@
 from app.exceptions import InventoryException
 from app.exceptions import ValidationException
 from app.logging import threadctx
+from app.models import Host
+from app.queue.event_producer import EventProducer
 from app.queue.events import EventType
 from app.queue.host_mq import IngressMessageConsumer
 from app.queue.host_mq import SystemProfileMessageConsumer
@@ -27,6 +34,7 @@
 from app.utils import Tag
 from lib.host_repository import AddHostResult
 from tests.helpers.db_utils import create_reference_host_in_db
+from tests.helpers.db_utils import minimal_db_host
 from tests.helpers.mq_utils import FakeMessage
 from tests.helpers.mq_utils import assert_mq_host_data
 from tests.helpers.mq_utils import expected_headers
@@ -40,6 +48,7 @@
 from tests.helpers.test_utils import USER_IDENTITY
 from tests.helpers.test_utils import YUM_REPO2
 from tests.helpers.test_utils import base_host
+from tests.helpers.test_utils import generate_random_string
 from tests.helpers.test_utils import generate_uuid
 from tests.helpers.test_utils import get_encoded_idstr
 from tests.helpers.test_utils import get_platform_metadata
@@ -1959,6 +1968,151 @@ def test_batch_mq_add_host_operations(mocker, event_producer, flask_app):
     assert write_batch_patch.call_count == 3
 
 
+def test_batch_mq_do_dedup_within_batch(
+    mocker: MockerFixture,
+    event_producer: EventProducer,
+    flask_app: FlaskApp,
+    db_get_hosts_by_subman_id: Callable[[str], list[Host]],
+):
+    """This test sends 2 kafka payloads for the same host in the same MQ batch and checks that
+    HBI applies dedup logic for messages within the same batch and doesn't create duplicates.
+    """
+    subman_id = generate_uuid()
+    host1 = minimal_host(
+        subscription_manager_id=subman_id,
+        fqdn=generate_random_string(),
+    ).data()
+    host2 = minimal_host(
+        subscription_manager_id=subman_id,
+        satellite_id=generate_uuid(),
+    ).data()
+
+    msgs = [json.dumps(wrap_message(host, "add_host", get_platform_metadata())) for host in (host1, host2)]
+
+    # Patch batch settings in inventory_config()
+    mocker.patch(
+        "app.queue.host_mq.inventory_config",
+        return_value=SimpleNamespace(
+            mq_db_batch_max_messages=2,
+            mq_db_batch_max_seconds=1,
+            culling_stale_warning_offset_delta=1,
+            culling_culled_offset_delta=1,
+        ),
+    )
+
+    fake_consumer = mocker.Mock(**{"consume.side_effect": [[FakeMessage(message=msg) for msg in msgs]]})
+
+    consumer = IngressMessageConsumer(fake_consumer, flask_app, event_producer, mocker.Mock())
+    consumer.event_loop(mocker.Mock(side_effect=(False, True)))
+
+    # Check that only 1 host was created and it contains data from both messages
+    db_hosts = db_get_hosts_by_subman_id(subman_id)
+    assert len(db_hosts) == 1
+    assert db_hosts[0].canonical_facts["fqdn"] == host1["fqdn"]
+    assert db_hosts[0].canonical_facts["satellite_id"] == host2["satellite_id"]
+
+
+def test_batch_mq_two_different_hosts(
+    mocker: MockerFixture,
+    event_producer: EventProducer,
+    flask_app: FlaskApp,
+    db_get_hosts_by_display_name: Callable[[str], list[Host]],
+):
+    """This test sends 2 kafka payloads for different hosts in one MQ batch and checks that
+    2 different hosts are created.
+    """
+    display_name = generate_random_string()
+    host1 = minimal_host(
+        subscription_manager_id=generate_uuid(),
+        display_name=display_name,
+    ).data()
+    host2 = minimal_host(
+        subscription_manager_id=generate_uuid(),
+        display_name=display_name,
+    ).data()
+
+    msgs = [json.dumps(wrap_message(host, "add_host", get_platform_metadata())) for host in (host1, host2)]
+
+    # Patch batch settings in inventory_config()
+    mocker.patch(
+        "app.queue.host_mq.inventory_config",
+        return_value=SimpleNamespace(
+            mq_db_batch_max_messages=2,
+            mq_db_batch_max_seconds=1,
+            culling_stale_warning_offset_delta=1,
+            culling_culled_offset_delta=1,
+        ),
+    )
+
+    fake_consumer = mocker.Mock(**{"consume.side_effect": [[FakeMessage(message=msg) for msg in msgs]]})
+
+    consumer = IngressMessageConsumer(fake_consumer, flask_app, event_producer, mocker.Mock())
+    consumer.event_loop(mocker.Mock(side_effect=(False, True)))
+
+    # Check that 2 different hosts have the same display_name
+    db_hosts = db_get_hosts_by_display_name(display_name)
+    assert len(db_hosts) == 2
+    assert db_hosts[0].id != db_hosts[1].id
+    assert (
+        db_hosts[0].canonical_facts["subscription_manager_id"]
+        != db_hosts[1].canonical_facts["subscription_manager_id"]
+    )
+
+
+def test_batch_mq_do_dedup_in_db(
+    mocker: MockerFixture,
+    event_producer: EventProducer,
+    flask_app: FlaskApp,
+    db_get_hosts_by_subman_id: Callable[[str], list[Host]],
+    db_get_hosts_by_display_name: Callable[[str], list[Host]],
+    db_create_host: Callable[..., Host],
+):
+    """This test sends 2 kafka payloads for different hosts in one MQ batch and checks that
+    HBI falls back to do dedup on hosts in the DB if it can't find the host in already processed
+    messages within the same batch.
+    """
+    subman_id = generate_uuid()
+    display_name = generate_random_string()
+    existing_host_data = minimal_db_host(
+        canonical_facts={"subscription_manager_id": subman_id, "fqdn": generate_random_string()},
+        display_name=display_name,
+    )
+    existing_host = db_create_host(host=existing_host_data)
+
+    msg_host1 = minimal_host(subscription_manager_id=generate_uuid(), display_name=display_name).data()
+    msg_host2 = minimal_host(
+        subscription_manager_id=subman_id, satellite_id=generate_uuid(), display_name=display_name
+    ).data()
+
+    msgs = [json.dumps(wrap_message(host, "add_host", get_platform_metadata())) for host in (msg_host1, msg_host2)]
+
+    # Patch batch settings in inventory_config()
+    mocker.patch(
+        "app.queue.host_mq.inventory_config",
+        return_value=SimpleNamespace(
+            mq_db_batch_max_messages=2,
+            mq_db_batch_max_seconds=1,
+            culling_stale_warning_offset_delta=1,
+            culling_culled_offset_delta=1,
+        ),
+    )
+
+    fake_consumer = mocker.Mock(**{"consume.side_effect": [[FakeMessage(message=msg) for msg in msgs]]})
+
+    consumer = IngressMessageConsumer(fake_consumer, flask_app, event_producer, mocker.Mock())
+    consumer.event_loop(mocker.Mock(side_effect=(False, True)))
+
+    # Check that 2 different hosts have the same display_name
+    db_hosts = db_get_hosts_by_display_name(display_name)
+    assert len(db_hosts) == 2
+
+    # Check that there is still only 1 host in DB with the same subman_id and it was properly updated
+    db_hosts = db_get_hosts_by_subman_id(subman_id)
+    assert len(db_hosts) == 1
+    assert db_hosts[0].canonical_facts["fqdn"] == existing_host.canonical_facts["fqdn"]
+    assert db_hosts[0].canonical_facts["satellite_id"] == msg_host2["satellite_id"]
+
+
 def test_batch_mq_header_request_id_updates(mocker, flask_app):
     # Verifies that when messages are sent as part of the same batch,
     # the request_id used in the header is updated correctly.


Reviews:
user: login: thearifismail
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-14T21:33:26Z
user: login: thearifismail
body: @fstavela, IMO typing of input params and return types should be separated from the checking for processed hosts. This will help readers focus more on the core subject of this PR.

Not done with review.
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-14T21:41:38Z
user: login: fstavela
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-15T12:45:42Z
user: login: fstavela
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-15T13:16:35Z
user: login: kruai
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-15T15:02:00Z
user: login: kruai
body: Apart from my suggestion PR (https://github.com/fstavela/insights-host-inventory/pull/16), I have one other suggestion for organization. The logic seems sound though.
I'm a bit concerned about how this will affect MQ performance, but I'm not sure how big of an impact it'll have. We should keep an eye on it after deployment.
state: CHANGES_REQUESTED
author_association: COLLABORATOR
submitted_at: 2025-05-15T18:42:46Z
user: login: fstavela
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-16T10:24:37Z
user: login: sourcery-ai[bot]
body: Hey @fstavela - I've reviewed your changes - here's some feedback:

- Extract the repeated “if from_hosts … else …” in-memory vs DB lookup logic into a shared helper to reduce duplication in `_find_host_by_elevated_ids` and `find_host_by_multiple_canonical_facts`.
- Update the signature of `matches_at_least_one_canonical_fact_filter_in_memory` to return `bool` instead of `BooleanClauseList` to match its implementation.
- Since `host_dedup_processing_time` now covers both DB and in-memory lookups, consider splitting or labeling the metric to distinguish the two paths for more accurate dedup performance tracking.

<details>
<summary>Here's what I looked at during the review</summary>

- 🟡 **General issues**: 2 issues found
- 🟢 **Testing**: all looks good
- 🟢 **Complexity**: all looks good
- 🟢 **Documentation**: all looks good
</details>

***

<details>
<summary>Sourcery is free for open source - if you like our reviews please consider sharing them ✨</summary>

- [X](https://twitter.com/intent/tweet?text=I%20just%20got%20an%20instant%20code%20review%20from%20%40SourceryAI%2C%20and%20it%20was%20brilliant%21%20It%27s%20free%20for%20open%20source%20and%20has%20a%20free%20trial%20for%20private%20code.%20Check%20it%20out%20https%3A//sourcery.ai)
- [Mastodon](https://mastodon.social/share?text=I%20just%20got%20an%20instant%20code%20review%20from%20%40SourceryAI%2C%20and%20it%20was%20brilliant%21%20It%27s%20free%20for%20open%20source%20and%20has%20a%20free%20trial%20for%20private%20code.%20Check%20it%20out%20https%3A//sourcery.ai)
- [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https://sourcery.ai)
- [Facebook](https://www.facebook.com/sharer/sharer.php?u=https://sourcery.ai)

</details>

<sub>
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
</sub>
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-16T15:05:49Z
user: login: kruai
body: I think this logic is sound. Would be worth addressing a couple of sourcery's comments, but I think it's good beyond that
state: DISMISSED
author_association: COLLABORATOR
submitted_at: 2025-05-16T17:01:02Z
user: login: fstavela
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-19T07:42:27Z
user: login: sourcery-ai[bot]
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-19T07:42:44Z
user: login: fstavela
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-19T07:51:26Z
user: login: fstavela
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-19T07:52:23Z
user: login: kruai
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-05-19T12:17:34Z
user: login: jpramos123
body: Logic looks solid. I have only one question when I was testing it, see below.

A suggestion for (maybe this) and future PRs. Continue to use conventional commits, but
break the commits into multiple atomic commits, instead of putting everything into one single big commit. This makes things easier to review, specially a big PR. 
I would break the commits for this PR into:

1. chore: Adjust type hints
2. fix: correctly process multiple...
3. chore: add metrics
4. tests: add tests

Thanks

state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-19T17:37:23Z
user: login: fstavela
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-20T10:03:44Z
user: login: jpramos123
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-21T12:12:43Z

Review comments:
diff_hunk: @@ -317,7 +318,7 @@ def rbac_permissions_filter(rbac_filter: dict) -> list:
     return _query_filter
 
 
-def update_query_for_owner_id(identity, query):
+def update_query_for_owner_id(identity: Identity, query: Query) -> Query:
     # kafka based requests have dummy identity for working around the identity requirement for CRUD operations
     logger.debug("identity auth type: %s", identity.auth_type)
     if identity and identity.identity_type == IdentityType.SYSTEM:
path: api/filtering/db_filters.py
user: login: thearifismail
body: This `identity` check should be before the `logger.debug()` because if identity is `null` then the logger.debug line will throw exception and this line will be skipped.
created_at: 2025-05-14T19:52:38Z
author_association: CONTRIBUTOR
diff_hunk: @@ -59,7 +65,11 @@
 
 
 def add_host(
-    input_host: Host, identity: Identity, update_system_profile: bool = True, operation_args: dict | None = None
+    input_host: Host,
+    identity: Identity,
+    update_system_profile: bool = True,
+    operation_args: dict | None = None,
path: lib/host_repository.py
user: login: thearifismail
body: what are operation_args?  How an operation_args object looks like?  An example as comment will be helpful
created_at: 2025-05-14T20:25:13Z
author_association: CONTRIBUTOR
diff_hunk: @@ -317,7 +318,7 @@ def rbac_permissions_filter(rbac_filter: dict) -> list:
     return _query_filter
 
 
-def update_query_for_owner_id(identity, query):
+def update_query_for_owner_id(identity: Identity, query: Query) -> Query:
     # kafka based requests have dummy identity for working around the identity requirement for CRUD operations
     logger.debug("identity auth type: %s", identity.auth_type)
     if identity and identity.identity_type == IdentityType.SYSTEM:
path: api/filtering/db_filters.py
user: login: fstavela
body: This wasn't changed in this PR, but I agree, so I updated it.
created_at: 2025-05-15T12:45:42Z
author_association: CONTRIBUTOR
diff_hunk: @@ -59,7 +65,11 @@
 
 
 def add_host(
-    input_host: Host, identity: Identity, update_system_profile: bool = True, operation_args: dict | None = None
+    input_host: Host,
+    identity: Identity,
+    update_system_profile: bool = True,
+    operation_args: dict | None = None,
path: lib/host_repository.py
user: login: fstavela
body: `operation_args` are documented here: https://inscope.corp.redhat.com/docs/default/Component/consoledot-pages/services/inventory/#expected-message-format

I added a comment about this to the docstring of this function.
created_at: 2025-05-15T13:16:34Z
author_association: CONTRIBUTOR
diff_hunk: @@ -312,7 +331,15 @@ def process_message(self, host_data, platform_metadata, operation_args=None):
                 input_host = _set_owner(input_host, identity)
 
             log_add_host_attempt(logger, input_host, sp_fields_to_log, identity)
-            host_row, add_result = host_repository.add_host(input_host, identity, operation_args=operation_args)
+            existing_host = None
+            if processed_rows:
+                processed_hosts = [result.host_row for result in processed_rows]
+                existing_host = host_repository.find_existing_host(
+                    identity, input_host.canonical_facts, processed_hosts
+                )
path: app/queue/host_mq.py
user: login: kruai
body: I think this logic should be contained within `add_host()` to make the dedup process a bit more clear. That way we can more clearly see that it looks for a host in memory first, and then looks in the DB.
created_at: 2025-05-15T15:01:59Z
author_association: COLLABORATOR
diff_hunk: @@ -312,7 +331,15 @@ def process_message(self, host_data, platform_metadata, operation_args=None):
                 input_host = _set_owner(input_host, identity)
 
             log_add_host_attempt(logger, input_host, sp_fields_to_log, identity)
-            host_row, add_result = host_repository.add_host(input_host, identity, operation_args=operation_args)
+            existing_host = None
+            if processed_rows:
+                processed_hosts = [result.host_row for result in processed_rows]
+                existing_host = host_repository.find_existing_host(
+                    identity, input_host.canonical_facts, processed_hosts
+                )
path: app/queue/host_mq.py
user: login: kruai
body: I think this logic should be contained within `add_host()` to make the dedup process a bit more clear. That way we can more clearly see that it looks for a host in memory first, and then looks in the DB.
created_at: 2025-05-15T18:40:32Z
author_association: COLLABORATOR
diff_hunk: @@ -312,7 +331,15 @@ def process_message(self, host_data, platform_metadata, operation_args=None):
                 input_host = _set_owner(input_host, identity)
 
             log_add_host_attempt(logger, input_host, sp_fields_to_log, identity)
-            host_row, add_result = host_repository.add_host(input_host, identity, operation_args=operation_args)
+            existing_host = None
+            if processed_rows:
+                processed_hosts = [result.host_row for result in processed_rows]
+                existing_host = host_repository.find_existing_host(
+                    identity, input_host.canonical_facts, processed_hosts
+                )
path: app/queue/host_mq.py
user: login: fstavela
body: I agree, I updated the PR. Please check if it looks good now.
created_at: 2025-05-16T10:24:37Z
author_association: CONTRIBUTOR
diff_hunk: @@ -131,13 +143,13 @@ def process_message(self, *args, **kwargs):
     def handle_message(self, *args, **kwargs) -> OperationResult | None:
         raise NotImplementedError("Not implemented in the HBIMessageConsumerBase class")
 
-    def post_process_rows(self, processed_rows: list[OperationResult]):
+    def post_process_rows(self):
path: app/queue/host_mq.py
user: login: sourcery-ai[bot]
body: **issue (bug_risk):** The post_process_rows method signature now differs from the base class.

The base class defines post_process_rows(processed_rows), but subclasses now drop the parameter and use self.processed_rows. This signature mismatch can break polymorphic usage. Either restore the parameter or document this change.
created_at: 2025-05-16T15:05:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -280,25 +292,30 @@ def handle_message(self, message) -> OperationResult:
                 )
                 raise
 
-    def post_process_rows(self, processed_rows: list[OperationResult]) -> None:
+    def post_process_rows(self) -> None:
         try:
-            if len(processed_rows) > 0:
+            if len(self.processed_rows) > 0:
                 db.session.commit()
                 # The above session is automatically committed or rolled back.
                 # Now we need to send out messages for the batch of hosts we just processed.
-                write_message_batch(self.event_producer, self.notification_event_producer, processed_rows)
+                write_message_batch(self.event_producer, self.notification_event_producer, self.processed_rows)
 
         except StaleDataError as exc:
-            metrics.ingress_message_handler_failure.inc(amount=len(processed_rows))
+            metrics.ingress_message_handler_failure.inc(amount=len(self.processed_rows))
             logger.error(
-                f"Session data is stale; failed to commit data from {len(processed_rows)} payloads.",
+                f"Session data is stale; failed to commit data from {len(self.processed_rows)} payloads.",
                 exc_info=exc,
             )
             db.session.rollback()
 
 
 class IngressMessageConsumer(HostMessageConsumer):
-    def process_message(self, host_data, platform_metadata, operation_args=None):
+    def process_message(
path: app/queue/host_mq.py
user: login: sourcery-ai[bot]
body: **nitpick:** The process_message return type is now a tuple, but the docstring is missing.

Add documentation for the four-element tuple return type in the process_message docstring to clarify its structure.
created_at: 2025-05-16T15:05:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -104,7 +126,9 @@ def find_existing_host_by_id(identity: Identity, host_id: str) -> Host | None:
 
 
 @metrics.find_host_using_elevated_ids.time()
-def _find_host_by_elevated_ids(identity: Identity, canonical_facts: dict) -> Host | None:
+def _find_host_by_elevated_ids(
path: lib/host_repository.py
user: login: sourcery-ai[bot]
body: **issue (code-quality):** Low code quality found in \_find\_host\_by\_elevated\_ids - 22% ([`low-code-quality`](https://docs.sourcery.ai/Reference/Default-Rules/comments/low-code-quality/))

<br/><details><summary>Explanation</summary>The quality score for this function is below the quality threshold of 25%.
This score is a combination of the method length, cognitive complexity and working memory.

How can you solve this?

It might be worth refactoring this function to make it shorter and more readable.

- Reduce the function length by extracting pieces of functionality out into
  their own functions. This is the most important thing you can do - ideally a
  function should be less than 10 lines.
- Reduce nesting, perhaps by introducing guard clauses to return early.
- Ensure that variables are tightly scoped, so that code using related concepts
  sits together within the function rather than being scattered.</details>
created_at: 2025-05-16T15:05:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -131,13 +143,13 @@ def process_message(self, *args, **kwargs):
     def handle_message(self, *args, **kwargs) -> OperationResult | None:
         raise NotImplementedError("Not implemented in the HBIMessageConsumerBase class")
 
-    def post_process_rows(self, processed_rows: list[OperationResult]):
+    def post_process_rows(self):
path: app/queue/host_mq.py
user: login: fstavela
body: Not true, base class also dropped the parameter, all definitions of `post_process_rows` are without any parameter.
created_at: 2025-05-19T07:42:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -131,13 +143,13 @@ def process_message(self, *args, **kwargs):
     def handle_message(self, *args, **kwargs) -> OperationResult | None:
         raise NotImplementedError("Not implemented in the HBIMessageConsumerBase class")
 
-    def post_process_rows(self, processed_rows: list[OperationResult]):
+    def post_process_rows(self):
path: app/queue/host_mq.py
user: login: sourcery-ai[bot]
body: You’re right—I was looking at an outdated snippet. The base class and all overrides now consistently drop the `processed_rows` parameter, so there’s no longer a mismatch. I’ll retract that comment. Sorry for the noise!
created_at: 2025-05-19T07:42:44Z
author_association: CONTRIBUTOR
diff_hunk: @@ -280,25 +292,30 @@ def handle_message(self, message) -> OperationResult:
                 )
                 raise
 
-    def post_process_rows(self, processed_rows: list[OperationResult]) -> None:
+    def post_process_rows(self) -> None:
         try:
-            if len(processed_rows) > 0:
+            if len(self.processed_rows) > 0:
                 db.session.commit()
                 # The above session is automatically committed or rolled back.
                 # Now we need to send out messages for the batch of hosts we just processed.
-                write_message_batch(self.event_producer, self.notification_event_producer, processed_rows)
+                write_message_batch(self.event_producer, self.notification_event_producer, self.processed_rows)
 
         except StaleDataError as exc:
-            metrics.ingress_message_handler_failure.inc(amount=len(processed_rows))
+            metrics.ingress_message_handler_failure.inc(amount=len(self.processed_rows))
             logger.error(
-                f"Session data is stale; failed to commit data from {len(processed_rows)} payloads.",
+                f"Session data is stale; failed to commit data from {len(self.processed_rows)} payloads.",
                 exc_info=exc,
             )
             db.session.rollback()
 
 
 class IngressMessageConsumer(HostMessageConsumer):
-    def process_message(self, host_data, platform_metadata, operation_args=None):
+    def process_message(
path: app/queue/host_mq.py
user: login: fstavela
body: Done
created_at: 2025-05-19T07:51:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -104,7 +126,9 @@ def find_existing_host_by_id(identity: Identity, host_id: str) -> Host | None:
 
 
 @metrics.find_host_using_elevated_ids.time()
-def _find_host_by_elevated_ids(identity: Identity, canonical_facts: dict) -> Host | None:
+def _find_host_by_elevated_ids(
path: lib/host_repository.py
user: login: fstavela
body: Should be fixed now that I extracted `_find_host_by_multiple_facts_in_db_or_in_memory` from this function
created_at: 2025-05-19T07:52:23Z
author_association: CONTRIBUTOR
diff_hunk: @@ -158,7 +170,7 @@ def event_loop(self, interrupt):
                             logger.debug("Message received")
 
                             try:
-                                processed_rows.append(self.handle_message(msg.value()))
+                                self.processed_rows.append(self.handle_message(msg.value()))
path: app/queue/host_mq.py
user: login: jpramos123
body: Made some tests locally, I was not able to populate this class attribute.

What may I be doing wrong?
created_at: 2025-05-19T17:24:23Z
author_association: CONTRIBUTOR
diff_hunk: @@ -158,7 +170,7 @@ def event_loop(self, interrupt):
                             logger.debug("Message received")
 
                             try:
-                                processed_rows.append(self.handle_message(msg.value()))
+                                self.processed_rows.append(self.handle_message(msg.value()))
path: app/queue/host_mq.py
user: login: fstavela
body: @jpramos123 Can you elaborate? Can you share your tests? What were you not able to populate? Did you overwrite the default `MQ_DB_BATCH_MAX_MESSAGES=1` to a higher number?
created_at: 2025-05-20T10:03:44Z
author_association: CONTRIBUTOR

PR comments:
user: login: sourcery-ai[bot]
created_at: 2025-05-13T13:54:27Z
body: <!-- Generated by sourcery-ai[bot]: start review_guide -->

## Reviewer's Guide

This PR refactors host deduplication to correctly handle multiple messages for the same host within one MQ batch by maintaining an in-memory list of processed hosts and filtering against it before falling back to database queries. It updates function signatures to accept optional host lists, implements pure-Python filters, and adjusts the MQ consumer loop to track and propagate processed rows.

### File-Level Changes

| Change | Details | Files |
| ------ | ------- | ----- |
| Introduce in-memory host lookup to avoid duplicates within the same batch | <ul><li>Add `from_hosts`/`existing_hosts` parameters to `find_existing_host` and related internal methods</li><li>Implement `multiple_canonical_facts_host_query_in_memory`, `contains_no_incorrect_facts_filter_in_memory`, and `matches_at_least_one_canonical_fact_filter_in_memory`</li><li>Branch logic in `_find_host_by_elevated_ids` and `find_host_by_multiple_canonical_facts` to use in-memory filters before DB queries</li></ul> | `lib/host_repository.py` |
| Track and propagate processed rows through the MQ consumer | <ul><li>Add `processed_rows` list attribute to `HBIMessageConsumerBase` and reset it each batch</li><li>Append messages to `self.processed_rows` in `event_loop`</li><li>Pass `processed_hosts` from `self.processed_rows` into `add_host` calls</li></ul> | `app/queue/host_mq.py` |
| Refine event batching and message writing behavior | <ul><li>Commit the DB session once per batch in `post_process_rows`</li><li>Use `write_message_batch` with the collected `processed_rows`</li><li>Validate non-null `event_type` and `platform_metadata` in `write_add_update_event_message`</li></ul> | `app/queue/host_mq.py`<br/>`app/queue/events.py` |
| Clean up obsolete queries and add type annotations | <ul><li>Remove unused `single_canonical_fact_host_query`</li><li>Update signature of `update_query_for_owner_id` and add return type</li><li>Add type hints to multiple functions, especially in MQ consumers and filters</li></ul> | `lib/host_repository.py`<br/>`api/filtering/db_filters.py`<br/>`app/queue/host_mq.py`<br/>`app/queue/mq_common.py` |
| Align tests and fixtures with new dedup logic and signatures | <ul><li>Add fixtures for fetching hosts by subscription ID/display name</li><li>Update tests to check in-batch dedup and DB fallback</li><li>Adjust signatures in test helpers for `minimal_host`, `minimal_db_host`, and `flask_app`</li></ul> | `tests/test_host_mq_service.py`<br/>`tests/fixtures/db_fixtures.py`<br/>`tests/fixtures/app_fixtures.py`<br/>`tests/helpers/db_utils.py` |

---

<details>
<summary>Tips and commands</summary>

#### Interacting with Sourcery

- **Trigger a new review:** Comment `@sourcery-ai review` on the pull request.
- **Continue discussions:** Reply directly to Sourcery's review comments.
- **Generate a GitHub issue from a review comment:** Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with `@sourcery-ai issue` to create an issue from it.
- **Generate a pull request title:** Write `@sourcery-ai` anywhere in the pull
  request title to generate a title at any time. You can also comment
  `@sourcery-ai title` on the pull request to (re-)generate the title at any time.
- **Generate a pull request summary:** Write `@sourcery-ai summary` anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment `@sourcery-ai summary` on the pull request to
  (re-)generate the summary at any time.
- **Generate reviewer's guide:** Comment `@sourcery-ai guide` on the pull
  request to (re-)generate the reviewer's guide at any time.
- **Resolve all Sourcery comments:** Comment `@sourcery-ai resolve` on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
- **Dismiss all Sourcery reviews:** Comment `@sourcery-ai dismiss` on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  `@sourcery-ai review` to trigger a new review!

#### Customizing Your Experience

Access your [dashboard](https://app.sourcery.ai) to:
- Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
- Change the review language.
- Add, remove or edit custom review instructions.
- Adjust other review settings.

#### Getting Help

- [Contact our support team](mailto:support@sourcery.ai) for questions or feedback.
- Visit our [documentation](https://docs.sourcery.ai) for detailed guides and information.
- Keep in touch with the Sourcery team by following us on [X/Twitter](https://x.com/SourceryAI), [LinkedIn](https://www.linkedin.com/company/sourcery-ai/) or [GitHub](https://github.com/sourcery-ai).

</details>

<!-- Generated by sourcery-ai[bot]: end review_guide -->
author_association: CONTRIBUTOR
user: login: fstavela
created_at: 2025-05-14T07:09:21Z
body: /retest
author_association: CONTRIBUTOR
user: login: thearifismail
created_at: 2025-05-14T19:24:15Z
body: IMO, In memory function should not use metrics.  The only effect of `in-memory function` on metrics should be decrease in the number of new hosts creations
author_association: CONTRIBUTOR
user: login: fstavela
created_at: 2025-05-15T13:22:15Z
body: > @fstavela, IMO typing of input params and return types should be separated from the checking for processed hosts. This will help readers focus more on the core subject of this PR.
> 
> Not done with review.

I added the type hints only to the functions that are related to the MQ processing and dedup, because that's what's being changed in this PR. Without the type hints, it's quite difficult for me to navigate the code, as I'm not very familiar with it yet. With the type hints, I know immediately what I can do with the parameters and what I can expect. They make the code more readable in my opinion.

I think that we could have a separate PR and a tech debt task for adding the type annotations, but on the other hand, it's easier to make small incremental changes rather than trying to type-annotate everything at once. So I think it's good to add the missing type annotations to the functions we are working with when we are making some changes.
author_association: CONTRIBUTOR
user: login: kruai
created_at: 2025-05-15T14:36:21Z
body: I'm sifting through the add_host logic making sure it works for various scenarios, but I first have a suggestion for processed_rows: https://github.com/fstavela/insights-host-inventory/pull/16
author_association: COLLABORATOR
user: login: fstavela
created_at: 2025-05-16T14:57:58Z
body: /retest
author_association: CONTRIBUTOR
user: login: fstavela
created_at: 2025-05-19T08:29:33Z
body: /retest
author_association: CONTRIBUTOR
user: login: jpramos123
created_at: 2025-05-19T17:39:16Z
body: > > @fstavela, IMO typing of input params and return types should be separated from the checking for processed hosts. This will help readers focus more on the core subject of this PR.
> > Not done with review.
> 
> I added the type hints only to the functions that are related to the MQ processing and dedup, because that's what's being changed in this PR. Without the type hints, it's quite difficult for me to navigate the code, as I'm not very familiar with it yet. With the type hints, I know immediately what I can do with the parameters and what I can expect. They make the code more readable in my opinion.
> 
> I think that we could have a separate PR and a tech debt task for adding the type annotations, but on the other hand, it's easier to make small incremental changes rather than trying to type-annotate everything at once. So I think it's good to add the missing type annotations to the functions we are working with when we are making some changes.

As I suggested, you may do it in the same PR, but in different commits! That makes things easier for the reviewers, as we can choose the commit, and check just the changes for that commit!
author_association: CONTRIBUTOR
user: login: fstavela
created_at: 2025-05-20T10:11:23Z
body: > Logic looks solid. I have only one question when I was testing it, see below.
> 
> A suggestion for (maybe this) and future PRs. Continue to use conventional commits, but break the commits into multiple atomic commits, instead of putting everything into one single big commit. This makes things easier to review, specially a big PR. I would break the commits for this PR into:
> 
>     1. chore: Adjust type hints
> 
>     2. fix: correctly process multiple...
> 
>     3. chore: add metrics
> 
>     4. tests: add tests
> 
> 
> Thanks

I can't really separate the type hints into their own commit. I'm not adding these type hints separately; I'm adding them "on the go" together with other code changes when I'm making them. I never know beforehand what functions need to be changed before I start working on my task, so I don't know where I'll need the type hints, so I don't think creating a separate commit for the type hints is possible.

But, regarding the other 3 commits, this is exactly how I had it initially. However, because these commits would be eventually squashed anyway (when the PR is merged), I merged them together to make the rebase conflicts easier to resolve. But I didn't realize that having them split would help with reviews, so I'll keep them separated next time. Thanks for the suggestion.
author_association: CONTRIBUTOR
user: login: fstavela
created_at: 2025-05-20T10:11:57Z
body: > > > @fstavela, IMO typing of input params and return types should be separated from the checking for processed hosts. This will help readers focus more on the core subject of this PR.
> > > Not done with review.
> > 
> > 
> > I added the type hints only to the functions that are related to the MQ processing and dedup, because that's what's being changed in this PR. Without the type hints, it's quite difficult for me to navigate the code, as I'm not very familiar with it yet. With the type hints, I know immediately what I can do with the parameters and what I can expect. They make the code more readable in my opinion.
> > I think that we could have a separate PR and a tech debt task for adding the type annotations, but on the other hand, it's easier to make small incremental changes rather than trying to type-annotate everything at once. So I think it's good to add the missing type annotations to the functions we are working with when we are making some changes.
> 
> As I suggested, you may do it in the same PR, but in different commits! That makes things easier for the reviewers, as we can choose the commit, and check just the changes for that commit!

Answered in the other response: https://github.com/RedHatInsights/insights-host-inventory/pull/2535#issuecomment-2893788536
author_association: CONTRIBUTOR
user: login: fstavela
created_at: 2025-05-21T07:30:37Z
body: /retest
author_association: CONTRIBUTOR
user: login: jpramos123
created_at: 2025-05-21T12:10:59Z
body: > > Logic looks solid. I have only one question when I was testing it, see below.
> 
> > 
> 
> > A suggestion for (maybe this) and future PRs. Continue to use conventional commits, but break the commits into multiple atomic commits, instead of putting everything into one single big commit. This makes things easier to review, specially a big PR. I would break the commits for this PR into:
> 
> > 
> 
> >     1. chore: Adjust type hints
> 
> > 
> 
> >     2. fix: correctly process multiple...
> 
> > 
> 
> >     3. chore: add metrics
> 
> > 
> 
> >     4. tests: add tests
> 
> > 
> 
> > 
> 
> > Thanks
> 
> 
> 
> I can't really separate the type hints into their own commit. I'm not adding these type hints separately; I'm adding them "on the go" together with other code changes when I'm making them. I never know beforehand what functions need to be changed before I start working on my task, so I don't know where I'll need the type hints, so I don't think creating a separate commit for the type hints is possible.
> 
> 
> 
> But, regarding the other 3 commits, this is exactly how I had it initially. However, because these commits would be eventually squashed anyway (when the PR is merged), I merged them together to make the rebase conflicts easier to resolve. But I didn't realize that having them split would help with reviews, so I'll keep them separated next time. Thanks for the suggestion.


You can use `git add -p` to add separate parts of your code, and commit them separately. This is a good way to split your commit into contexts! Keep this in mind for a next PR!
author_association: CONTRIBUTOR