Repository: NCSU-High-Powered-Rocketry-Club/AirbrakesV2
Pull request #168: Imu optimization
Author: login: JacksonElia
Created: 2025-03-03T02:55:40Z | Merged: 2025-03-03T03:52:30Z

When making either a RawDataPacket or EstimatedDataPacket we were unnecessarily checking all of the fields for both of them. Instead, I made it so it only checks for raw fields when populating a RawDataPacket, and estimated fields when populating an EstimatedDataPacket.

Additionally, I thought of a further optimization that I might try if @harshil21 thinks it would actually help: Once we populate a field, we know for that data packet, we won't have to populate it again. So what if we made a list of fields that we check to add to a data packet, and as we add them to said data packet, we remove them from the list so that the fields at the end will have to do fewer comparisons?


Diff:
diff --git a/airbrakes/hardware/imu.py b/airbrakes/hardware/imu.py
index cf854811..77b342d9 100644
--- a/airbrakes/hardware/imu.py
+++ b/airbrakes/hardware/imu.py
@@ -100,7 +100,7 @@ def _fetch_data_loop(self, port: str) -> None:  # pragma: no cover
         # - using 2 inner loops depending on the type of packet, this reduces the number of `if`
         # checks the interpreter has to do
         # - Using integers for comparison instead of strings (O(1) vs O(n) complexity)
-        # - Checking for raw data packets first, since that is 2x the frequency of estimated packets
+        # - Checking for raw data packets or estimated data packets, not both
         # - Using msgspec to serialize and deserialize the packets, which is faster than pickle
         # - High priority for the process
         while self.is_running:
@@ -118,131 +118,136 @@ def _fetch_data_loop(self, port: str) -> None:  # pragma: no cover
                 # Initialize packet with the timestamp, determines if the packet is raw or estimated
                 if descriptor_set == RAW_DESCRIPTOR_SET:
                     imu_data_packet = RawDataPacket(timestamp)
+                    # Iterate through each data point in the packet.
+                    for data_point in packet.data():
+                        # Extract the channel name of the data point.
+                        qualifier = data_point.qualifier()
+                        field_name = data_point.field()
+
+                        if field_name == SCALED_ACCEL_FIELD:
+                            # Scaled acceleration data
+                            if qualifier == X_QUALIFIER:
+                                imu_data_packet.scaledAccelX = data_point.as_float()
+                            elif qualifier == Y_QUALIFIER:
+                                imu_data_packet.scaledAccelY = data_point.as_float()
+                            elif qualifier == Z_QUALIFIER:
+                                imu_data_packet.scaledAccelZ = data_point.as_float()
+
+                        elif field_name == SCALED_GYRO_FIELD:
+                            # Scaled gyroscope data
+                            if qualifier == X_QUALIFIER:
+                                imu_data_packet.scaledGyroX = data_point.as_float()
+                            elif qualifier == Y_QUALIFIER:
+                                imu_data_packet.scaledGyroY = data_point.as_float()
+                            elif qualifier == Z_QUALIFIER:
+                                imu_data_packet.scaledGyroZ = data_point.as_float()
+
+                        elif field_name == DELTA_VEL_FIELD:
+                            # Delta velocity (change in velocity)
+                            if qualifier == X_QUALIFIER:
+                                imu_data_packet.deltaVelX = data_point.as_float()
+                            elif qualifier == Y_QUALIFIER:
+                                imu_data_packet.deltaVelY = data_point.as_float()
+                            elif qualifier == Z_QUALIFIER:
+                                imu_data_packet.deltaVelZ = data_point.as_float()
+
+                        elif field_name == DELTA_THETA_FIELD:
+                            # Delta theta (change in orientation)
+                            if qualifier == X_QUALIFIER:
+                                imu_data_packet.deltaThetaX = data_point.as_float()
+                            elif qualifier == Y_QUALIFIER:
+                                imu_data_packet.deltaThetaY = data_point.as_float()
+                            elif qualifier == Z_QUALIFIER:
+                                imu_data_packet.deltaThetaZ = data_point.as_float()
+
+                        elif (
+                            field_name == SCALED_AMBIENT_PRESSURE_FIELD
+                            and qualifier == AMBIENT_PRESSURE_QUALIFIER
+                        ):
+                            # Scaled ambient pressure data
+                            imu_data_packet.scaledAmbientPressure = data_point.as_float()
+
                 elif descriptor_set == ESTIMATED_DESCRIPTOR_SET:
                     imu_data_packet = EstimatedDataPacket(timestamp)
+                    for data_point in packet.data():
+                        # Extract the channel name of the data point.
+                        qualifier = data_point.qualifier()
+                        field_name = data_point.field()
+
+                        if (
+                            field_name == EST_PRESSURE_ALT_FIELD
+                            and qualifier == PRESSURE_ALT_QUALIFIER
+                        ):
+                            # Estimated pressure altitude
+                            imu_data_packet.estPressureAlt = data_point.as_float()
+
+                        elif (
+                            field_name == EST_ORIENT_QUATERNION_FIELD
+                            and qualifier == ATTITUDE_UNCERT_QUALIFIER
+                        ):
+                            # Estimated orientation quaternion
+                            matrix = data_point.as_Matrix()
+                            # The imu sends the quaternions as a matrix, so we have to unpack it
+                            imu_data_packet.estOrientQuaternionW = matrix.as_floatAt(0, 0)
+                            imu_data_packet.estOrientQuaternionX = matrix.as_floatAt(0, 1)
+                            imu_data_packet.estOrientQuaternionY = matrix.as_floatAt(0, 2)
+                            imu_data_packet.estOrientQuaternionZ = matrix.as_floatAt(0, 3)
+
+                        elif (
+                            field_name == EST_ATTITUDE_UNCERT_FIELD
+                            and qualifier == ATTITUDE_UNCERT_QUALIFIER
+                        ):
+                            # Estimated attitude uncertainty quaternion
+                            matrix = data_point.as_Matrix()
+                            imu_data_packet.estAttitudeUncertQuaternionW = matrix.as_floatAt(0, 0)
+                            imu_data_packet.estAttitudeUncertQuaternionX = matrix.as_floatAt(0, 1)
+                            imu_data_packet.estAttitudeUncertQuaternionY = matrix.as_floatAt(0, 2)
+                            imu_data_packet.estAttitudeUncertQuaternionZ = matrix.as_floatAt(0, 3)
+
+                        elif field_name == EST_ANGULAR_RATE_FIELD:
+                            # Estimated angular rate
+                            if qualifier == X_QUALIFIER:
+                                imu_data_packet.estAngularRateX = data_point.as_float()
+                            elif qualifier == Y_QUALIFIER:
+                                imu_data_packet.estAngularRateY = data_point.as_float()
+                            elif qualifier == Z_QUALIFIER:
+                                imu_data_packet.estAngularRateZ = data_point.as_float()
+
+                        elif field_name == EST_COMPENSATED_ACCEL_FIELD:
+                            # Estimated compensated acceleration
+                            if qualifier == X_QUALIFIER:
+                                imu_data_packet.estCompensatedAccelX = data_point.as_float()
+                            elif qualifier == Y_QUALIFIER:
+                                imu_data_packet.estCompensatedAccelY = data_point.as_float()
+                            elif qualifier == Z_QUALIFIER:
+                                imu_data_packet.estCompensatedAccelZ = data_point.as_float()
+
+                        elif field_name == EST_LINEAR_ACCEL_FIELD:
+                            # Estimated linear acceleration
+                            if qualifier == X_QUALIFIER:
+                                imu_data_packet.estLinearAccelX = data_point.as_float()
+                            elif qualifier == Y_QUALIFIER:
+                                imu_data_packet.estLinearAccelY = data_point.as_float()
+                            elif qualifier == Z_QUALIFIER:
+                                imu_data_packet.estLinearAccelZ = data_point.as_float()
+
+                        elif field_name == EST_GRAVITY_VECTOR_FIELD:
+                            # Estimated gravity vector
+                            if qualifier == X_QUALIFIER:
+                                imu_data_packet.estGravityVectorX = data_point.as_float()
+                            elif qualifier == Y_QUALIFIER:
+                                imu_data_packet.estGravityVectorY = data_point.as_float()
+                            elif qualifier == Z_QUALIFIER:
+                                imu_data_packet.estGravityVectorZ = data_point.as_float()
+
+                        # Check if the data point is invalid and update the invalid fields list.
+                        if not data_point.valid():
+                            if imu_data_packet.invalid_fields is None:
+                                imu_data_packet.invalid_fields = []
+                            imu_data_packet.invalid_fields.append(data_point.channelName())
                 else:
                     continue  # We never actually reach here, but keeping it just in case
 
-                # Iterate through each data point in the packet.
-                for data_point in packet.data():
-                    # Extract the channel name of the data point.
-                    qualifier = data_point.qualifier()
-                    field_name = data_point.field()
-
-                    if field_name == DELTA_THETA_FIELD:
-                        # Delta theta (change in orientation)
-                        if qualifier == X_QUALIFIER:
-                            imu_data_packet.deltaThetaX = data_point.as_float()
-                        elif qualifier == Y_QUALIFIER:
-                            imu_data_packet.deltaThetaY = data_point.as_float()
-                        elif qualifier == Z_QUALIFIER:
-                            imu_data_packet.deltaThetaZ = data_point.as_float()
-
-                    elif field_name == DELTA_VEL_FIELD:
-                        # Delta velocity (change in velocity)
-                        if qualifier == X_QUALIFIER:
-                            imu_data_packet.deltaVelX = data_point.as_float()
-                        elif qualifier == Y_QUALIFIER:
-                            imu_data_packet.deltaVelY = data_point.as_float()
-                        elif qualifier == Z_QUALIFIER:
-                            imu_data_packet.deltaVelZ = data_point.as_float()
-
-                    elif field_name == SCALED_ACCEL_FIELD:
-                        # Scaled acceleration data
-                        if qualifier == X_QUALIFIER:
-                            imu_data_packet.scaledAccelX = data_point.as_float()
-                        elif qualifier == Y_QUALIFIER:
-                            imu_data_packet.scaledAccelY = data_point.as_float()
-                        elif qualifier == Z_QUALIFIER:
-                            imu_data_packet.scaledAccelZ = data_point.as_float()
-
-                    elif (
-                        field_name == SCALED_AMBIENT_PRESSURE_FIELD
-                        and qualifier == AMBIENT_PRESSURE_QUALIFIER
-                    ):
-                        # Scaled ambient pressure data
-                        imu_data_packet.scaledAmbientPressure = data_point.as_float()
-
-                    elif field_name == SCALED_GYRO_FIELD:
-                        # Scaled gyroscope data
-                        if qualifier == X_QUALIFIER:
-                            imu_data_packet.scaledGyroX = data_point.as_float()
-                        elif qualifier == Y_QUALIFIER:
-                            imu_data_packet.scaledGyroY = data_point.as_float()
-                        elif qualifier == Z_QUALIFIER:
-                            imu_data_packet.scaledGyroZ = data_point.as_float()
-
-                    elif field_name == EST_ANGULAR_RATE_FIELD:
-                        # Estimated angular rate
-                        if qualifier == X_QUALIFIER:
-                            imu_data_packet.estAngularRateX = data_point.as_float()
-                        elif qualifier == Y_QUALIFIER:
-                            imu_data_packet.estAngularRateY = data_point.as_float()
-                        elif qualifier == Z_QUALIFIER:
-                            imu_data_packet.estAngularRateZ = data_point.as_float()
-
-                    elif (
-                        field_name == EST_ATTITUDE_UNCERT_FIELD
-                        and qualifier == ATTITUDE_UNCERT_QUALIFIER
-                    ):
-                        # Estimated attitude uncertainty quaternion
-                        matrix = data_point.as_Matrix()
-                        imu_data_packet.estAttitudeUncertQuaternionW = matrix.as_floatAt(0, 0)
-                        imu_data_packet.estAttitudeUncertQuaternionX = matrix.as_floatAt(0, 1)
-                        imu_data_packet.estAttitudeUncertQuaternionY = matrix.as_floatAt(0, 2)
-                        imu_data_packet.estAttitudeUncertQuaternionZ = matrix.as_floatAt(0, 3)
-
-                    elif field_name == EST_COMPENSATED_ACCEL_FIELD:
-                        # Estimated compensated acceleration
-                        if qualifier == X_QUALIFIER:
-                            imu_data_packet.estCompensatedAccelX = data_point.as_float()
-                        elif qualifier == Y_QUALIFIER:
-                            imu_data_packet.estCompensatedAccelY = data_point.as_float()
-                        elif qualifier == Z_QUALIFIER:
-                            imu_data_packet.estCompensatedAccelZ = data_point.as_float()
-
-                    elif field_name == EST_GRAVITY_VECTOR_FIELD:
-                        # Estimated gravity vector
-                        if qualifier == X_QUALIFIER:
-                            imu_data_packet.estGravityVectorX = data_point.as_float()
-                        elif qualifier == Y_QUALIFIER:
-                            imu_data_packet.estGravityVectorY = data_point.as_float()
-                        elif qualifier == Z_QUALIFIER:
-                            imu_data_packet.estGravityVectorZ = data_point.as_float()
-
-                    elif field_name == EST_LINEAR_ACCEL_FIELD:
-                        # Estimated linear acceleration
-                        if qualifier == X_QUALIFIER:
-                            imu_data_packet.estLinearAccelX = data_point.as_float()
-                        elif qualifier == Y_QUALIFIER:
-                            imu_data_packet.estLinearAccelY = data_point.as_float()
-                        elif qualifier == Z_QUALIFIER:
-                            imu_data_packet.estLinearAccelZ = data_point.as_float()
-
-                    elif (
-                        field_name == EST_ORIENT_QUATERNION_FIELD
-                        and qualifier == ATTITUDE_UNCERT_QUALIFIER
-                    ):
-                        # Estimated orientation quaternion
-                        matrix = data_point.as_Matrix()
-                        # The imu sends the quaternions as a matrix, so we have to unpack it
-                        imu_data_packet.estOrientQuaternionW = matrix.as_floatAt(0, 0)
-                        imu_data_packet.estOrientQuaternionX = matrix.as_floatAt(0, 1)
-                        imu_data_packet.estOrientQuaternionY = matrix.as_floatAt(0, 2)
-                        imu_data_packet.estOrientQuaternionZ = matrix.as_floatAt(0, 3)
-
-                    elif (
-                        field_name == EST_PRESSURE_ALT_FIELD and qualifier == PRESSURE_ALT_QUALIFIER
-                    ):
-                        # Estimated pressure altitude
-                        imu_data_packet.estPressureAlt = data_point.as_float()
-
-                    # Check if the data point is invalid and update the invalid fields list.
-                    if not data_point.valid():
-                        if imu_data_packet.invalid_fields is None:
-                            imu_data_packet.invalid_fields = []
-                        imu_data_packet.invalid_fields.append(data_point.channelName())
-
                     # Unused channels include: `gpsCorrelTimestamp(Tow,WeekNum,Flags)`,
                     # `estFilterGpsTimeTow`, `estFilterGpsTimeWeekNum`. But we
                     # can't exclude it from the IMU settings cause it says it's not recommended
diff --git a/airbrakes/telemetry/packets/imu_data_packet.py b/airbrakes/telemetry/packets/imu_data_packet.py
index 5601d175..299dbfde 100644
--- a/airbrakes/telemetry/packets/imu_data_packet.py
+++ b/airbrakes/telemetry/packets/imu_data_packet.py
@@ -48,11 +48,11 @@ class EstimatedDataPacket(IMUDataPacket):
     It contains a timestamp and the estimated values of the relevant data points.
     """
 
+    estPressureAlt: float | None = None
     estOrientQuaternionW: float | None = None
     estOrientQuaternionX: float | None = None
     estOrientQuaternionY: float | None = None
     estOrientQuaternionZ: float | None = None
-    estPressureAlt: float | None = None
     estAttitudeUncertQuaternionW: float | None = None
     estAttitudeUncertQuaternionX: float | None = None
     estAttitudeUncertQuaternionY: float | None = None
diff --git a/airbrakes/telemetry/packets/logger_data_packet.py b/airbrakes/telemetry/packets/logger_data_packet.py
index 0fbc0b5a..66261b19 100644
--- a/airbrakes/telemetry/packets/logger_data_packet.py
+++ b/airbrakes/telemetry/packets/logger_data_packet.py
@@ -37,11 +37,11 @@ class LoggerDataPacket(msgspec.Struct, array_like=True, kw_only=True):
     scaledAmbientPressure: float | None = None
 
     # Estimated Data Packet Fields
+    estPressureAlt: float | None = None
     estOrientQuaternionW: float | None = None
     estOrientQuaternionX: float | None = None
     estOrientQuaternionY: float | None = None
     estOrientQuaternionZ: float | None = None
-    estPressureAlt: float | None = None
     estAttitudeUncertQuaternionW: float | None = None
     estAttitudeUncertQuaternionX: float | None = None
     estAttitudeUncertQuaternionY: float | None = None


Reviews:
user: login: harshil21
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-03T03:07:26Z
user: login: JacksonElia
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-03T03:14:47Z

Review comments:
diff_hunk: @@ -48,11 +48,11 @@ class EstimatedDataPacket(IMUDataPacket):
     It contains a timestamp and the estimated values of the relevant data points.
     """
 
+    estPressureAlt: float | None = None
     estOrientQuaternionW: float | None = None
     estOrientQuaternionX: float | None = None
     estOrientQuaternionY: float | None = None
     estOrientQuaternionZ: float | None = None
-    estPressureAlt: float | None = None
     estAttitudeUncertQuaternionW: float | None = None
path: airbrakes/telemetry/packets/imu_data_packet.py
user: login: harshil21
body: how does this affect performance
created_at: 2025-03-03T03:07:26Z
author_association: MEMBER
diff_hunk: @@ -48,11 +48,11 @@ class EstimatedDataPacket(IMUDataPacket):
     It contains a timestamp and the estimated values of the relevant data points.
     """
 
+    estPressureAlt: float | None = None
     estOrientQuaternionW: float | None = None
     estOrientQuaternionX: float | None = None
     estOrientQuaternionY: float | None = None
     estOrientQuaternionZ: float | None = None
-    estPressureAlt: float | None = None
     estAttitudeUncertQuaternionW: float | None = None
path: airbrakes/telemetry/packets/imu_data_packet.py
user: login: JacksonElia
body: it doesn't, I just thought it made more sense to have our more important field higher up like that, rather than sandwiched between quaternions. 
created_at: 2025-03-03T03:14:47Z
author_association: MEMBER

PR comments:
user: login: harshil21
created_at: 2025-03-03T03:00:49Z
body: > we were unnecessarily checking all of the fields for both of them. Instead, I made it so it only checks for raw fields when populating a RawDataPacket, and estimated fields when populating an EstimatedDataPacket.

it was initially like this and you reverted it. What changed? 

>  So what if we made a list of fields that we check to add to a data packet, and as we add them to said data packet, we remove them from the list so that the fields at the end will have to do fewer comparisons?

how do you want to check from that list? a `getattr` call? That would be more expensive than just iterating through the conditions. 
author_association: MEMBER
user: login: harshil21
created_at: 2025-03-03T03:06:03Z
body: I also thought of using a mapping with the keys as the field names, and then the values as function to call which would set the attribute. This would be constant time. However I think the overhead of calling a function would offset that unfortunately.
author_association: MEMBER
user: login: JacksonElia
created_at: 2025-03-03T03:11:47Z
body: >it was initially like this and you reverted it. What changed?

I never reverted it? I remember telling you and craig after doing these optimizations that you should do this

>how do you want to check from that list? a getattr call? That would be more expensive than just iterating through the conditions.

Yeah I'm not really sure. ~Maybe we could do something with a dict but it might end up not being as fast.~ Okay after thinking it through more, you're right. The only thing I can think of is have booleans for each field and once we check it, we make those booleans false, so our if statements look like `if field_bool and field_name == field`

~This might be faster because comparing strings are kinda expensive~ Nvm we are comparing ints, so I guess its fine

author_association: MEMBER
user: login: harshil21
created_at: 2025-03-03T03:47:50Z
body: > I never reverted it? I remember telling you and craig after doing these optimizations that you should do this

https://github.com/NCSU-High-Powered-Rocketry-Club/AirbrakesV2/pull/146/commits/1f2e423db0d03e19661c4e8f9d286f2b008dfd78
author_association: MEMBER
user: login: harshil21
created_at: 2025-03-03T03:48:07Z
body: I can merge after physically testing it on the pi.
author_association: MEMBER
user: login: JacksonElia
created_at: 2025-03-03T03:52:06Z
body: Lol I guess I accidentally did when switching over to the constants, woops
author_association: MEMBER
user: login: JacksonElia
created_at: 2025-03-03T03:52:25Z
body: Nah lets just merge it now, we can test everything on decent-changes before merging
author_association: MEMBER