Repository: autowarefoundation/autoware_universe
Pull request #10284: feat(lidar_centerpoint): add diagnostics for processing time
Author: login: a-maumau
Labels: name: component:perception
description: Advanced sensor data processing and environment understanding. (auto-assigned)
name: run:build-and-test-differential
description: Mark to enable build-and-test-differential workflow. (used-by-ci)
Created: 2025-03-17T09:56:15Z | Merged: 2025-04-03T09:11:12Z

## Description

This PR adds diagnostics for processing time.

If the processing time exceeds `max_allowed_processing_time_ms`, a warning topic will be published.

Consecutive delays will be also checked.
If the difference of two timestamp (current and last in-time processed) is larger than `max_acceptable_consecutive_delay_ms`, it will determine as an error state.

If the new processing time is below the `max_allowed_processing_time_ms`, it will update the timestamp of last normally processed case.

## Related links

**Parent Issue:**

- Link

<!-- ⬇️🟢
**Private Links:**

- [CompanyName internal link]()
⬆️🟢 -->

## How was this PR tested?
Tested by checking the rqt_runtime_monitor.

Run rqt_runtime_monitor by
```sh
# optional
# ros2 run autoware_diagnostic_graph_utils converter_node

ros2 run rqt_runtime_monitor rqt_runtime_monitor
```
https://github.com/user-attachments/assets/4b816d36-ff8d-4777-bb32-395350a1c338


(above video was created in sample-rosbag with very small `max_allowed_processing_time_ms` == 2ms for demonstration)

## Notes for reviewers


None.

## Interface changes

None.

<!-- ⬇️🔴

### Topic changes

#### Additions and removals

| Change type   | Topic Type      | Topic Name    | Message Type        | Description       |
|:--------------|:----------------|:--------------|:--------------------|:------------------|
| Added/Removed | Pub/Sub/Srv/Cli | `/topic_name` | `std_msgs/String`   | Topic description |

#### Modifications

| Version | Topic Type      | Topic Name        | Message Type        | Description       |
|:--------|:----------------|:------------------|:--------------------|:------------------|
| Old     | Pub/Sub/Srv/Cli | `/old_topic_name` | `sensor_msgs/Image` | Topic description |
| New     | Pub/Sub/Srv/Cli | `/new_topic_name` | `sensor_msgs/Image` | Topic description |

### ROS Parameter Changes

#### Additions and removals

| Change type   | Parameter Name | Type     | Default Value | Description       |
|:--------------|:---------------|:---------|:--------------|:------------------|
| Added/Removed | `param_name`   | `double` | `1.0`         | Param description |

#### Modifications

| Version | Parameter Name   | Type     | Default Value | Description       |
|:--------|:-----------------|:---------|:--------------|:------------------|
| Old     | `old_param_name` | `double` | `1.0`         | Param description |
| New     | `new_param_name` | `double` | `1.0`         | Param description |

🔴⬆️ -->

## Effects on system behavior

None.



Diff:
diff --git a/perception/autoware_lidar_centerpoint/config/centerpoint_common.param.yaml b/perception/autoware_lidar_centerpoint/config/centerpoint_common.param.yaml
new file mode 100644
index 0000000000000..572f04055341e
--- /dev/null
+++ b/perception/autoware_lidar_centerpoint/config/centerpoint_common.param.yaml
@@ -0,0 +1,11 @@
+/**:
+  ros__parameters:
+    # When the processing time exceeds the max_allowed_processing_time,
+    # a warning will be triggered.
+    diagnostics:
+      max_allowed_processing_time_ms: 200.0 # in milliseconds
+      # If the warning is triggered and if a timestamp of last normal processing is more than
+      # max_acceptable_consecutive_delay_ms, an error diagnostic message will be sent.
+      max_acceptable_consecutive_delay_ms: 1000.0
+      # interval of diagnostic callback in milliseconds
+      validation_callback_interval_ms: 100.0 # in milliseconds
diff --git a/perception/autoware_lidar_centerpoint/include/autoware/lidar_centerpoint/node.hpp b/perception/autoware_lidar_centerpoint/include/autoware/lidar_centerpoint/node.hpp
index 673f4bbf71262..afbaf4c2dfd71 100644
--- a/perception/autoware_lidar_centerpoint/include/autoware/lidar_centerpoint/node.hpp
+++ b/perception/autoware_lidar_centerpoint/include/autoware/lidar_centerpoint/node.hpp
@@ -27,6 +27,7 @@
 #include <cuda_blackboard/cuda_blackboard_subscriber.hpp>
 #include <cuda_blackboard/cuda_pointcloud2.hpp>
 #include <cuda_blackboard/negotiated_types.hpp>
+#include <diagnostic_updater/diagnostic_updater.hpp>
 #include <rclcpp/rclcpp.hpp>
 
 #include <autoware_perception_msgs/msg/detected_object_kinematics.hpp>
@@ -34,6 +35,7 @@
 #include <autoware_perception_msgs/msg/object_classification.hpp>
 #include <autoware_perception_msgs/msg/shape.hpp>
 #include <sensor_msgs/msg/point_cloud2.hpp>
+#include <std_msgs/msg/header.hpp>
 
 #include <memory>
 #include <string>
@@ -50,6 +52,7 @@ class LidarCenterPointNode : public rclcpp::Node
 private:
   void pointCloudCallback(
     const std::shared_ptr<const cuda_blackboard::CudaPointCloud2> & input_pointcloud_msg);
+  void diagnoseProcessingTime(diagnostic_updater::DiagnosticStatusWrapper & stat);
 
   tf2_ros::Buffer tf_buffer_;
   tf2_ros::TransformListener tf_listener_{tf_buffer_};
@@ -62,11 +65,19 @@ class LidarCenterPointNode : public rclcpp::Node
   bool has_variance_{false};
   bool has_twist_{false};
 
+  // for diagnostics
+  double max_allowed_processing_time_ms_;
+  double max_acceptable_consecutive_delay_ms_;
+  // set as optional to avoid sending error diagnostics before the node starts processing
+  std::optional<double> last_processing_time_ms_;
+  std::optional<rclcpp::Time> last_in_time_processing_timestamp_;
+  diagnostic_updater::Updater diagnostic_processing_time_updater_{this};
+
   NonMaximumSuppression iou_bev_nms_;
   DetectionClassRemapper detection_class_remapper_;
 
   std::unique_ptr<CenterPointTRT> detector_ptr_{nullptr};
-  std::unique_ptr<autoware_utils::DiagnosticsInterface> diagnostics_interface_ptr_;
+  std::unique_ptr<autoware_utils::DiagnosticsInterface> diagnostics_centerpoint_trt_;
 
   // debugger
   std::unique_ptr<autoware_utils::StopWatch<std::chrono::milliseconds>> stop_watch_ptr_{nullptr};
diff --git a/perception/autoware_lidar_centerpoint/launch/lidar_centerpoint.launch.xml b/perception/autoware_lidar_centerpoint/launch/lidar_centerpoint.launch.xml
index af5b7a5b15033..52d185a07689e 100644
--- a/perception/autoware_lidar_centerpoint/launch/lidar_centerpoint.launch.xml
+++ b/perception/autoware_lidar_centerpoint/launch/lidar_centerpoint.launch.xml
@@ -8,6 +8,7 @@
   <arg name="model_param_path" default="$(find-pkg-share autoware_lidar_centerpoint)/config/$(var model_name).param.yaml"/>
   <arg name="ml_package_param_path" default="$(var model_path)/$(var model_name)_ml_package.param.yaml"/>
   <arg name="class_remapper_param_path" default="$(var model_path)/detection_class_remapper.param.yaml"/>
+  <arg name="common_param_path" default="$(find-pkg-share autoware_lidar_centerpoint)/config/centerpoint_common.param.yaml"/>
   <arg name="build_only" default="false" description="shutdown node after TensorRT engine file is built"/>
 
   <arg name="use_pointcloud_container" default="false" description="use pointcloud_container"/>
@@ -22,6 +23,7 @@
         <param from="$(var model_param_path)" allow_substs="true"/>
         <param from="$(var ml_package_param_path)" allow_substs="true"/>
         <param from="$(var class_remapper_param_path)"/>
+        <param from="$(var common_param_path)"/>
 
         <!-- This parameter shall NOT be included in param file. See also: https://github.com/autowarefoundation/autoware_universe/pull/6167 -->
         <param name="build_only" value="$(var build_only)"/>
@@ -36,6 +38,7 @@
       <param from="$(var model_param_path)" allow_substs="true"/>
       <param from="$(var ml_package_param_path)" allow_substs="true"/>
       <param from="$(var class_remapper_param_path)"/>
+      <param from="$(var common_param_path)"/>
 
       <!-- This parameter shall NOT be included in param file. See also: https://github.com/autowarefoundation/autoware_universe/pull/6167 -->
       <param name="build_only" value="$(var build_only)"/>
diff --git a/perception/autoware_lidar_centerpoint/schema/centerpoint_common.schema.json b/perception/autoware_lidar_centerpoint/schema/centerpoint_common.schema.json
new file mode 100644
index 0000000000000..f756724ad1e21
--- /dev/null
+++ b/perception/autoware_lidar_centerpoint/schema/centerpoint_common.schema.json
@@ -0,0 +1,49 @@
+{
+  "$schema": "http://json-schema.org/draft-07/schema#",
+  "title": "Parameters for Lidar Centerpoint Diagnostics",
+  "type": "object",
+  "definitions": {
+    "centerpoint_common": {
+      "type": "object",
+      "properties": {
+        "diagnostics": {
+          "type": "object",
+          "description": "Parameters for diagnostics.",
+          "properties": {
+            "max_allowed_processing_time_ms": {
+              "type": "number",
+              "description": "A threshold value for the allowed processing time. If the processing time exceeds this value, it will be considered a warning state. [ms]",
+              "default": 200.0,
+              "minimum": 0.0
+            },
+            "max_acceptable_consecutive_delay_ms": {
+              "type": "number",
+              "description": "A threshold value for the error state. If the duration since the last processing timestamp, which ended within max_allowed_processing_time_ms, exceeds this value, it will be considered an error state. [ms]",
+              "default": 1000.0,
+              "minimum": 0.0
+            },
+            "validation_callback_interval_ms": {
+              "type": "number",
+              "description": "An interval value for a timer callback that checks whether the current state meets the max_acceptable_consecutive_delay_ms condition. [ms]",
+              "default": 100.0,
+              "minimum": 1.0
+            }
+          }
+        }
+      },
+      "required": ["diagnostics"]
+    }
+  },
+  "properties": {
+    "/**": {
+      "type": "object",
+      "properties": {
+        "ros__parameters": {
+          "$ref": "#/definitions/centerpoint_common"
+        }
+      },
+      "required": ["ros__parameters"]
+    }
+  },
+  "required": ["/**"]
+}
diff --git a/perception/autoware_lidar_centerpoint/src/node.cpp b/perception/autoware_lidar_centerpoint/src/node.cpp
index 1952ab76fa111..6169c550bd5b6 100644
--- a/perception/autoware_lidar_centerpoint/src/node.cpp
+++ b/perception/autoware_lidar_centerpoint/src/node.cpp
@@ -106,9 +106,15 @@ LidarCenterPointNode::LidarCenterPointNode(const rclcpp::NodeOptions & node_opti
     circle_nms_dist_threshold, yaw_norm_thresholds, has_variance_);
   detector_ptr_ =
     std::make_unique<CenterPointTRT>(encoder_param, head_param, densification_param, config);
-  diagnostics_interface_ptr_ =
+  diagnostics_centerpoint_trt_ =
     std::make_unique<autoware_utils::DiagnosticsInterface>(this, "centerpoint_trt");
 
+  // diagnostics parameters
+  max_allowed_processing_time_ms_ =
+    declare_parameter<double>("diagnostics.max_allowed_processing_time_ms");
+  max_acceptable_consecutive_delay_ms_ =
+    declare_parameter<double>("diagnostics.max_acceptable_consecutive_delay_ms");
+
   pointcloud_sub_ =
     std::make_unique<cuda_blackboard::CudaBlackboardSubscriber<cuda_blackboard::CudaPointCloud2>>(
       *this, "~/input/pointcloud",
@@ -126,6 +132,18 @@ LidarCenterPointNode::LidarCenterPointNode(const rclcpp::NodeOptions & node_opti
     stop_watch_ptr_->tic("processing_time");
   }
 
+  if (stop_watch_ptr_) {
+    // processing time diagnostics
+    const double validation_callback_interval_ms =
+      declare_parameter<double>("diagnostics.validation_callback_interval_ms");
+
+    diagnostic_processing_time_updater_.setHardwareID(this->get_name());
+    diagnostic_processing_time_updater_.add(
+      "processing_time_status", this, &LidarCenterPointNode::diagnoseProcessingTime);
+    // msec -> sec
+    diagnostic_processing_time_updater_.setPeriod(validation_callback_interval_ms / 1e3);
+  }
+
   if (this->declare_parameter("build_only", false)) {
     RCLCPP_INFO(this->get_logger(), "TensorRT engine is built and shutdown node.");
     rclcpp::shutdown();
@@ -145,7 +163,7 @@ void LidarCenterPointNode::pointCloudCallback(
   if (stop_watch_ptr_) {
     stop_watch_ptr_->toc("processing_time", true);
   }
-  diagnostics_interface_ptr_->clear();
+  diagnostics_centerpoint_trt_->clear();
 
   std::vector<Box3D> det_boxes3d;
   bool is_num_pillars_within_range = true;
@@ -154,13 +172,13 @@ void LidarCenterPointNode::pointCloudCallback(
   if (!is_success) {
     return;
   }
-  diagnostics_interface_ptr_->add_key_value(
+  diagnostics_centerpoint_trt_->add_key_value(
     "is_num_pillars_within_range", is_num_pillars_within_range);
   if (!is_num_pillars_within_range) {
     std::stringstream message;
     message << "CenterPointTRT::detect: The actual number of pillars exceeds its maximum value, "
             << "which may limit the detection performance.";
-    diagnostics_interface_ptr_->update_level_and_message(
+    diagnostics_centerpoint_trt_->update_level_and_message(
       diagnostic_msgs::msg::DiagnosticStatus::WARN, message.str());
   }
 
@@ -182,7 +200,7 @@ void LidarCenterPointNode::pointCloudCallback(
     objects_pub_->publish(output_msg);
     published_time_publisher_->publish_if_subscribed(objects_pub_, output_msg.header.stamp);
   }
-  diagnostics_interface_ptr_->publish(input_pointcloud_msg->header.stamp);
+  diagnostics_centerpoint_trt_->publish(input_pointcloud_msg->header.stamp);
 
   // add processing time for debug
   if (debug_publisher_ptr_ && stop_watch_ptr_) {
@@ -199,9 +217,73 @@ void LidarCenterPointNode::pointCloudCallback(
       "debug/processing_time_ms", processing_time_ms);
     debug_publisher_ptr_->publish<autoware_internal_debug_msgs::msg::Float64Stamped>(
       "debug/pipeline_latency_ms", pipeline_latency_ms);
+
+    last_processing_time_ms_ = processing_time_ms;
   }
 }
 
+// Check the processing time and delayed timestamp
+// If the node is consistently delayed, publish an error diagnostic message
+void LidarCenterPointNode::diagnoseProcessingTime(
+  diagnostic_updater::DiagnosticStatusWrapper & stat)
+{
+  const rclcpp::Time timestamp_now = this->get_clock()->now();
+  diagnostic_msgs::msg::DiagnosticStatus::_level_type diag_level =
+    diagnostic_msgs::msg::DiagnosticStatus::OK;
+  std::stringstream message{"OK"};
+
+  // Check if the node has performed inference
+  if (last_processing_time_ms_) {
+    // check processing time is acceptable
+    if (last_processing_time_ms_ > max_allowed_processing_time_ms_) {
+      stat.add("is_processing_time_ms_in_expected_range", false);
+
+      message.clear();
+      message << "Processing time exceeds the acceptable limit of "
+              << max_allowed_processing_time_ms_ << " ms by "
+              << (last_processing_time_ms_.value() - max_allowed_processing_time_ms_) << " ms.";
+
+      // in case the processing starts with a delayed inference
+      if (!last_in_time_processing_timestamp_) {
+        last_in_time_processing_timestamp_ = timestamp_now;
+      }
+
+      diag_level = diagnostic_msgs::msg::DiagnosticStatus::WARN;
+    } else {
+      stat.add("is_processing_time_ms_in_expected_range", true);
+      last_in_time_processing_timestamp_ = timestamp_now;
+    }
+    stat.add("processing_time_ms", last_processing_time_ms_.value());
+
+    const double delayed_state_duration =
+      std::chrono::duration<double, std::milli>(
+        std::chrono::nanoseconds(
+          (timestamp_now - last_in_time_processing_timestamp_.value()).nanoseconds()))
+        .count();
+
+    // check consecutive delays
+    if (delayed_state_duration > max_acceptable_consecutive_delay_ms_) {
+      stat.add("is_consecutive_processing_delay_in_range", false);
+
+      message << " Processing delay has consecutively exceeded the acceptable limit continuously.";
+
+      diag_level = diagnostic_msgs::msg::DiagnosticStatus::ERROR;
+    } else {
+      stat.add("is_consecutive_processing_delay_in_range", true);
+    }
+    stat.add("consecutive_processing_delay_ms", delayed_state_duration);
+  } else {
+    stat.add("is_processing_time_ms_in_expected_range", true);
+    stat.add("processing_time_ms", 0.0);
+    stat.add("is_consecutive_processing_delay_in_range", true);
+    stat.add("consecutive_processing_delay_ms", 0.0);
+
+    message << "Waiting for the node to perform inference.";
+  }
+
+  stat.summary(diag_level, message.str());
+}
+
 }  // namespace autoware::lidar_centerpoint
 
 #include <rclcpp_components/register_node_macro.hpp>


Reviews:
user: login: technolojin
body: overall, the implementation looks good.

I suggested some renaming.
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T04:37:24Z
user: login: technolojin
body: LGTM
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-27T09:10:27Z
user: login: knzo25
body: LGTM
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-31T01:29:18Z
user: login: technolojin
body: I would like to request to implement to two types of diagnostics

1. number of voxel limit (already implemented)
2. processing time diags

when those are implemented, please let me know.
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-01T06:42:27Z
user: login: technolojin
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-03T08:03:20Z
user: login: a-maumau
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-03T08:39:26Z
user: login: technolojin
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-03T08:48:50Z
user: login: technolojin
body: LGTM
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-03T08:49:53Z
user: login: technolojin
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-03T08:51:57Z

Review comments:
diff_hunk: @@ -56,11 +58,19 @@ class LidarCenterPointNode : public rclcpp::Node
   bool has_variance_{false};
   bool has_twist_{false};
 
+  bool is_processing_delayed_{false};
+  double max_allowed_processing_time_ms_;
+  double max_acceptable_consecutive_delay_ms_;
+  // set optional to avoid sending the error diagnostics before node start processing
+  std::optional<rclcpp::Time> last_normal_processing_timestamp_;
path: perception/autoware_lidar_centerpoint/include/autoware/lidar_centerpoint/node.hpp
user: login: technolojin
body: ```suggestion
  std::optional<rclcpp::Time> last_in_time_processing_timestamp_;
```
created_at: 2025-03-25T04:20:44Z
author_association: CONTRIBUTOR
path: perception/autoware_lidar_centerpoint/config/centerpoint_diagnostics.param.yaml
user: login: technolojin
body: This configuration is for general node parameters. Therefore, it is proper decision to make a separated parameter file.

Suggestion:
`centerpoint_diagnostics.param.yaml` -> `centerpoint_node.param.yaml` or `centerpoint_common.param.yaml` 
created_at: 2025-03-25T04:29:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,10 @@
+/**:
+  ros__parameters:
+    # When the processing time exceeds the max_allowed_processing_time,
+    # a warning will be triggered.
+    max_allowed_processing_time_ms: 200.0 # in milliseconds
+    # If the warning is triggered and if a timestamp of last normal processing is more than
+    # max_acceptable_consecutive_delay_ms, an error diagnostic message will be sent.
+    max_acceptable_consecutive_delay_ms: 1000.0
+    # interval of diagnostic callback in milliseconds
+    validation_callback_interval_ms: 100 # in milliseconds, in integer
path: perception/autoware_lidar_centerpoint/config/centerpoint_diagnostics.param.yaml
user: login: technolojin
body: ```suggestion
    diagnostics:
      max_allowed_processing_time_ms: 200.0 # in milliseconds
      # If the warning is triggered and if a timestamp of last normal processing is more than
      # max_acceptable_consecutive_delay_ms, an error diagnostic message will be sent.
      max_acceptable_consecutive_delay_ms: 1000.0
      # interval of diagnostic callback in milliseconds
      validation_callback_interval_ms: 100 # in milliseconds, in integer
```
created_at: 2025-03-25T04:30:06Z
author_association: CONTRIBUTOR
diff_hunk: @@ -199,9 +217,73 @@ void LidarCenterPointNode::pointCloudCallback(
       "debug/processing_time_ms", processing_time_ms);
     debug_publisher_ptr_->publish<autoware_internal_debug_msgs::msg::Float64Stamped>(
       "debug/pipeline_latency_ms", pipeline_latency_ms);
+
+    last_processing_time_ms_ = processing_time_ms;
   }
 }
 
+// Check the processing time and delayed timestamp
+// If the node is consistently delayed, publish an error diagnostic message
+void LidarCenterPointNode::diagnoseProcessingTime(
+  diagnostic_updater::DiagnosticStatusWrapper & stat)
+{
+  const rclcpp::Time timestamp_now = this->get_clock()->now();
+  diagnostic_msgs::msg::DiagnosticStatus::_level_type diag_level =
+    diagnostic_msgs::msg::DiagnosticStatus::OK;
+  std::stringstream message{"OK"};
+
+  // check node has inferenced or not
+  if (last_processing_time_ms_) {
+    // check processing time is acceptable
+    if (last_processing_time_ms_ > max_allowed_processing_time_ms_) {
+      stat.add("is_processing_time_ms_in_expected_range", false);
+
+      message.clear();
+      message << "Processing time exceeds the acceptable limit of "
+              << max_allowed_processing_time_ms_ << " ms by "
+              << (last_processing_time_ms_.value() - max_allowed_processing_time_ms_) << " ms.";
+
+      // in case the processing starts with a delayed inference
+      if (!last_in_time_processing_timestamp_) {
+        last_in_time_processing_timestamp_ = timestamp_now;
+      }
+
+      diag_level = diagnostic_msgs::msg::DiagnosticStatus::WARN;
+    } else {
+      stat.add("is_processing_time_ms_in_expected_range", true);
+      last_in_time_processing_timestamp_ = timestamp_now;
+    }
+    stat.add("processing_time_ms", last_processing_time_ms_.value());
+
+    const double delayed_state_duration =
+      std::chrono::duration<double, std::milli>(
+        std::chrono::nanoseconds(
+          (timestamp_now - last_in_time_processing_timestamp_.value()).nanoseconds()))
+        .count();
+
+    // check consecutive delays
+    if (delayed_state_duration > max_acceptable_consecutive_delay_ms_) {
+      stat.add("is_consecutive_processing_delay_in_range", false);
+
+      message << " Processing delay has consecutively exceeded the acceptable limit continuously.";
+
+      diag_level = diagnostic_msgs::msg::DiagnosticStatus::ERROR;
+    } else {
+      stat.add("is_consecutive_processing_delay_in_range", true);
+    }
+    stat.add("consecutive_processing_delay_ms", delayed_state_duration);
+  } else {
+    stat.add("is_processing_time_ms_in_expected_range", true);
+    stat.add("processing_time_ms", 0.0);
+    stat.add("is_consecutive_processing_delay_in_range", true);
+    stat.add("consecutive_processing_delay_ms", 0.0);
path: perception/autoware_lidar_centerpoint/src/node.cpp
user: login: technolojin
body: May not needed, but acceptable.
created_at: 2025-04-03T08:03:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -199,9 +217,73 @@ void LidarCenterPointNode::pointCloudCallback(
       "debug/processing_time_ms", processing_time_ms);
     debug_publisher_ptr_->publish<autoware_internal_debug_msgs::msg::Float64Stamped>(
       "debug/pipeline_latency_ms", pipeline_latency_ms);
+
+    last_processing_time_ms_ = processing_time_ms;
   }
 }
 
+// Check the processing time and delayed timestamp
+// If the node is consistently delayed, publish an error diagnostic message
+void LidarCenterPointNode::diagnoseProcessingTime(
+  diagnostic_updater::DiagnosticStatusWrapper & stat)
+{
+  const rclcpp::Time timestamp_now = this->get_clock()->now();
+  diagnostic_msgs::msg::DiagnosticStatus::_level_type diag_level =
+    diagnostic_msgs::msg::DiagnosticStatus::OK;
+  std::stringstream message{"OK"};
+
+  // check node has inferenced or not
+  if (last_processing_time_ms_) {
+    // check processing time is acceptable
+    if (last_processing_time_ms_ > max_allowed_processing_time_ms_) {
+      stat.add("is_processing_time_ms_in_expected_range", false);
+
+      message.clear();
+      message << "Processing time exceeds the acceptable limit of "
+              << max_allowed_processing_time_ms_ << " ms by "
+              << (last_processing_time_ms_.value() - max_allowed_processing_time_ms_) << " ms.";
+
+      // in case the processing starts with a delayed inference
+      if (!last_in_time_processing_timestamp_) {
+        last_in_time_processing_timestamp_ = timestamp_now;
+      }
+
+      diag_level = diagnostic_msgs::msg::DiagnosticStatus::WARN;
+    } else {
+      stat.add("is_processing_time_ms_in_expected_range", true);
+      last_in_time_processing_timestamp_ = timestamp_now;
+    }
+    stat.add("processing_time_ms", last_processing_time_ms_.value());
+
+    const double delayed_state_duration =
+      std::chrono::duration<double, std::milli>(
+        std::chrono::nanoseconds(
+          (timestamp_now - last_in_time_processing_timestamp_.value()).nanoseconds()))
+        .count();
+
+    // check consecutive delays
+    if (delayed_state_duration > max_acceptable_consecutive_delay_ms_) {
+      stat.add("is_consecutive_processing_delay_in_range", false);
+
+      message << " Processing delay has consecutively exceeded the acceptable limit continuously.";
+
+      diag_level = diagnostic_msgs::msg::DiagnosticStatus::ERROR;
+    } else {
+      stat.add("is_consecutive_processing_delay_in_range", true);
+    }
+    stat.add("consecutive_processing_delay_ms", delayed_state_duration);
+  } else {
+    stat.add("is_processing_time_ms_in_expected_range", true);
+    stat.add("processing_time_ms", 0.0);
+    stat.add("is_consecutive_processing_delay_in_range", true);
+    stat.add("consecutive_processing_delay_ms", 0.0);
path: perception/autoware_lidar_centerpoint/src/node.cpp
user: login: a-maumau
body: I was wondering if it is OK to change fields depending on the state.
created_at: 2025-04-03T08:39:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -199,9 +217,73 @@ void LidarCenterPointNode::pointCloudCallback(
       "debug/processing_time_ms", processing_time_ms);
     debug_publisher_ptr_->publish<autoware_internal_debug_msgs::msg::Float64Stamped>(
       "debug/pipeline_latency_ms", pipeline_latency_ms);
+
+    last_processing_time_ms_ = processing_time_ms;
   }
 }
 
+// Check the processing time and delayed timestamp
+// If the node is consistently delayed, publish an error diagnostic message
+void LidarCenterPointNode::diagnoseProcessingTime(
+  diagnostic_updater::DiagnosticStatusWrapper & stat)
+{
+  const rclcpp::Time timestamp_now = this->get_clock()->now();
+  diagnostic_msgs::msg::DiagnosticStatus::_level_type diag_level =
+    diagnostic_msgs::msg::DiagnosticStatus::OK;
+  std::stringstream message{"OK"};
+
+  // check node has inferenced or not
+  if (last_processing_time_ms_) {
+    // check processing time is acceptable
+    if (last_processing_time_ms_ > max_allowed_processing_time_ms_) {
+      stat.add("is_processing_time_ms_in_expected_range", false);
+
+      message.clear();
+      message << "Processing time exceeds the acceptable limit of "
+              << max_allowed_processing_time_ms_ << " ms by "
+              << (last_processing_time_ms_.value() - max_allowed_processing_time_ms_) << " ms.";
+
+      // in case the processing starts with a delayed inference
+      if (!last_in_time_processing_timestamp_) {
+        last_in_time_processing_timestamp_ = timestamp_now;
+      }
+
+      diag_level = diagnostic_msgs::msg::DiagnosticStatus::WARN;
+    } else {
+      stat.add("is_processing_time_ms_in_expected_range", true);
+      last_in_time_processing_timestamp_ = timestamp_now;
+    }
+    stat.add("processing_time_ms", last_processing_time_ms_.value());
+
+    const double delayed_state_duration =
+      std::chrono::duration<double, std::milli>(
+        std::chrono::nanoseconds(
+          (timestamp_now - last_in_time_processing_timestamp_.value()).nanoseconds()))
+        .count();
+
+    // check consecutive delays
+    if (delayed_state_duration > max_acceptable_consecutive_delay_ms_) {
+      stat.add("is_consecutive_processing_delay_in_range", false);
+
+      message << " Processing delay has consecutively exceeded the acceptable limit continuously.";
+
+      diag_level = diagnostic_msgs::msg::DiagnosticStatus::ERROR;
+    } else {
+      stat.add("is_consecutive_processing_delay_in_range", true);
+    }
+    stat.add("consecutive_processing_delay_ms", delayed_state_duration);
+  } else {
+    stat.add("is_processing_time_ms_in_expected_range", true);
+    stat.add("processing_time_ms", 0.0);
+    stat.add("is_consecutive_processing_delay_in_range", true);
+    stat.add("consecutive_processing_delay_ms", 0.0);
path: perception/autoware_lidar_centerpoint/src/node.cpp
user: login: technolojin
body: In the very beginning, the field is empty (due to the diagnostic_updater implementation)
created_at: 2025-04-03T08:48:50Z
author_association: CONTRIBUTOR
diff_hunk: @@ -199,9 +217,73 @@ void LidarCenterPointNode::pointCloudCallback(
       "debug/processing_time_ms", processing_time_ms);
     debug_publisher_ptr_->publish<autoware_internal_debug_msgs::msg::Float64Stamped>(
       "debug/pipeline_latency_ms", pipeline_latency_ms);
+
+    last_processing_time_ms_ = processing_time_ms;
   }
 }
 
+// Check the processing time and delayed timestamp
+// If the node is consistently delayed, publish an error diagnostic message
+void LidarCenterPointNode::diagnoseProcessingTime(
+  diagnostic_updater::DiagnosticStatusWrapper & stat)
+{
+  const rclcpp::Time timestamp_now = this->get_clock()->now();
+  diagnostic_msgs::msg::DiagnosticStatus::_level_type diag_level =
+    diagnostic_msgs::msg::DiagnosticStatus::OK;
+  std::stringstream message{"OK"};
+
+  // check node has inferenced or not
path: perception/autoware_lidar_centerpoint/src/node.cpp
user: login: technolojin
body: ```suggestion
  // Check if the node has performed inference
```
created_at: 2025-04-03T08:51:57Z
author_association: CONTRIBUTOR

PR comments:
user: login: github-actions[bot]
created_at: 2025-03-17T09:56:29Z
body: Thank you for contributing to the Autoware project!

🚧 If your pull request is in progress, [switch it to draft mode](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/changing-the-stage-of-a-pull-request#converting-a-pull-request-to-a-draft).

Please ensure:
- You've checked our [contribution guidelines](https://autowarefoundation.github.io/autoware-documentation/main/contributing/).
- Your PR follows our [pull request guidelines](https://autowarefoundation.github.io/autoware-documentation/main/contributing/pull-request-guidelines/).
- All required CI checks pass before [marking the PR ready for review](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/changing-the-stage-of-a-pull-request#marking-a-pull-request-as-ready-for-review).
<!-- Sticky Pull Request Comment -->
author_association: NONE
user: login: codecov[bot]
created_at: 2025-03-26T00:44:06Z
body: ## [Codecov](https://app.codecov.io/gh/autowarefoundation/autoware_universe/pull/10284?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=autowarefoundation) Report
Attention: Patch coverage is `0%` with `41 lines` in your changes missing coverage. Please review.
> Project coverage is 25.43%. Comparing base [(`15641ff`)](https://app.codecov.io/gh/autowarefoundation/autoware_universe/commit/15641ff55ac4c073a1613a16a92196db94108296?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=autowarefoundation) to head [(`0fed2fa`)](https://app.codecov.io/gh/autowarefoundation/autoware_universe/commit/0fed2fa6e4274ae6de162783128ae4cc8e3a1ac9?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=autowarefoundation).
> Report is 4 commits behind head on main.

| [Files with missing lines](https://app.codecov.io/gh/autowarefoundation/autoware_universe/pull/10284?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=autowarefoundation) | Patch % | Lines |
|---|---|---|
| [perception/autoware\_lidar\_centerpoint/src/node.cpp](https://app.codecov.io/gh/autowarefoundation/autoware_universe/pull/10284?src=pr&el=tree&filepath=perception%2Fautoware_lidar_centerpoint%2Fsrc%2Fnode.cpp&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=autowarefoundation#diff-cGVyY2VwdGlvbi9hdXRvd2FyZV9saWRhcl9jZW50ZXJwb2ludC9zcmMvbm9kZS5jcHA=) | 0.00% | [41 Missing :warning: ](https://app.codecov.io/gh/autowarefoundation/autoware_universe/pull/10284?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=autowarefoundation) |

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main   #10284      +/-   ##
==========================================
+ Coverage   24.68%   25.43%   +0.75%     
==========================================
  Files        1313     1321       +8     
  Lines      102699   102711      +12     
  Branches    38994    39042      +48     
==========================================
+ Hits        25348    26125     +777     
+ Misses      74942    73883    -1059     
- Partials     2409     2703     +294     
```

| [Flag](https://app.codecov.io/gh/autowarefoundation/autoware_universe/pull/10284/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=autowarefoundation) | Coverage Δ | | *Carryforward flag |
|---|---|---|---|
| [differential](https://app.codecov.io/gh/autowarefoundation/autoware_universe/pull/10284/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=autowarefoundation) | `4.17% <ø> (?)` | | |
| [differential-cuda](https://app.codecov.io/gh/autowarefoundation/autoware_universe/pull/10284/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=autowarefoundation) | `6.31% <0.00%> (?)` | | |
| [total](https://app.codecov.io/gh/autowarefoundation/autoware_universe/pull/10284/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=autowarefoundation) | `25.45% <0.00%> (+0.77%)` | :arrow_up: | Carriedforward from [14ef798](https://app.codecov.io/gh/autowarefoundation/autoware_universe/commit/14ef798598ea531c0291aba3945726a4bf12ea3c?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=autowarefoundation) |

*This pull request uses carry forward flags. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=autowarefoundation) to find out more.

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/autowarefoundation/autoware_universe/pull/10284?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=autowarefoundation).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=autowarefoundation).

<details><summary> :rocket: New features to boost your workflow: </summary>

- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
author_association: NONE
user: login: a-maumau
created_at: 2025-03-27T01:50:02Z
body: @technolojin I agreed with your suggestions and applied in 934070c to db4d837 .
author_association: CONTRIBUTOR
user: login: a-maumau
created_at: 2025-03-27T06:40:28Z
body: I will create a different PR for the launcher related things.
author_association: CONTRIBUTOR
user: login: knzo25
created_at: 2025-03-28T02:50:15Z
body: @a-maumau 
Can you check why the CI/CD does not pass?
author_association: CONTRIBUTOR
user: login: a-maumau
created_at: 2025-03-28T07:54:49Z
body: I am not sure with the previous status, but currently it seems it has some issue with docker pull (code: TOOMANYREQUESTS)
author_association: CONTRIBUTOR
user: login: technolojin
created_at: 2025-03-31T01:05:02Z
body: clang-tidy was failed
```
private field 'is_processing_delayed_' is not used [clang-diagnostic-unused-private-field][0m
  bool is_processing_delayed_{false};
```
author_association: CONTRIBUTOR
user: login: a-maumau
created_at: 2025-04-02T06:12:35Z
body: @technolojin 
I done some refactor in ff8185d to reorganize and merge the diag. to publish in same interface.

Current implementation may have a little bit of downside.
Unlike in map_based_prediction: https://github.com/autowarefoundation/autoware_universe/blob/6fca99a2194c793079468e2fb982d48e6ba38ef4/perception/autoware_map_based_prediction/src/map_based_prediction_node.cpp#L736 , I implemented the diagnosis for consecutive delay in timer callback to ensure some temporal resolution, and this give some unliked output of diagnostics which processing time is set `0.0` since we can not compute in the timer callback.

I wonder this is an issue for later processings like analysis or logging.

![sample_diag](https://github.com/user-attachments/assets/0dda455e-25eb-47af-9311-7308b811963b)

author_association: CONTRIBUTOR
user: login: a-maumau
created_at: 2025-04-03T07:37:18Z
body: Sorry for my non concrete impelmentation that making many fix after the first review.

I change the code to use `diagnostic_updater` to diagnose the processing time status.
Since node's callback is currently set in mutually exclusive callback, this should make the timer callback suspended till the `pointCloudCallback` finishes, so the "ensure some temporal resolution" things I mentioned does not make much sense.

I made the code more simple.

updated the sample image in the top PR comment.
author_association: CONTRIBUTOR
user: login: technolojin
created_at: 2025-05-08T05:29:57Z
body: launcher config https://github.com/autowarefoundation/autoware_launch/pull/1377
author_association: CONTRIBUTOR