An apparatus for implementing semantic mapping, the apparatus comprising: a dynamic vision sensor configured to output an asynchronous stream of sensor event data; a complementary metal-oxide-semiconductor (CMOS) image sensor configured to output frames of image data; an inertial measurement unit; a sensor controller and scheduler (540, 700) configured to operate intermittently, or only as needed, the CMOS image sensor based on a current value of one or more CMOS control factors, the CMOS image sensor supplementing the image data obtained from the DVS sensor; a processor; and a memory containing instructions, which when executed by the processor, cause the apparatus to: generate a semantic segmentation of a time-stamped frame, the time-stamped frame based on one or more of an output of the CMOS image sensor, or a synthesized event frame based on an output from the DVS and an output from the IMU over a time interval, wherein the semantic segmentation comprises a semantic label associated with a region of the time-stamped frame, receive, from a visual-inertial simultaneous location and mapping (SLAM) pipeline, pose data of the apparatus and point cloud data, wherein the pose data and the point cloud data are associated with the time-stamped frame, determine, based on the semantic segmentation, the pose data and the point cloud data, a simplified object representation in a coordinate space, wherein the simplified object representation comprises a higher-level geometric primitive associated with the semantic label, and update a semantic map based on the simplified object representation. a dynamic vision sensor configured to output an asynchronous stream of sensor event data; a complementary metal-oxide-semiconductor (CMOS) image sensor configured to output frames of image data; an inertial measurement unit; a sensor controller and scheduler (540, 700) configured to operate intermittently, or only as needed, the CMOS image sensor based on a current value of one or more CMOS control factors, the CMOS image sensor supplementing the image data obtained from the DVS sensor; a processor; and a memory containing instructions, which when executed by the processor, cause the apparatus to: generate a semantic segmentation of a time-stamped frame, the time-stamped frame based on one or more of an output of the CMOS image sensor, or a synthesized event frame based on an output from the DVS and an output from the IMU over a time interval, wherein the semantic segmentation comprises a semantic label associated with a region of the time-stamped frame, receive, from a visual-inertial simultaneous location and mapping (SLAM) pipeline, pose data of the apparatus and point cloud data, wherein the pose data and the point cloud data are associated with the time-stamped frame, determine, based on the semantic segmentation, the pose data and the point cloud data, a simplified object representation in a coordinate space, wherein the simplified object representation comprises a higher-level geometric primitive associated with the semantic label, and update a semantic map based on the simplified object representation. generate a semantic segmentation of a time-stamped frame, the time-stamped frame based on one or more of an output of the CMOS image sensor, or a synthesized event frame based on an output from the DVS and an output from the IMU over a time interval, wherein the semantic segmentation comprises a semantic label associated with a region of the time-stamped frame, receive, from a visual-inertial simultaneous location and mapping (SLAM) pipeline, pose data of the apparatus and point cloud data, wherein the pose data and the point cloud data are associated with the time-stamped frame, determine, based on the semantic segmentation, the pose data and the point cloud data, a simplified object representation in a coordinate space, wherein the simplified object representation comprises a higher-level geometric primitive associated with the semantic label, and update a semantic map based on the simplified object representation.