Repository: carla-simulator/carla (language: python)

Issue:
Different on total point counts
<!--

Thanks for contributing to CARLA!

If you are asking a question please make sure your question was not asked before
by searching among the existing issues. Also make sure you have read our
documentation and FAQ at http://carla.readthedocs.io.

If your question is about creating content (assets, levels) you'll most likely
have all the info you need in the Unreal Engine's documentation
https://docs.unrealengine.com

Please, keep the threads focused and single-themed. Make them easy to find
for everyone.

If you are reporting an issue, please provide the CARLA version number you are
using and your Platform/OS.

-->
Hi,
I'm using C++ example with Carla 0.9.11 to get total points from lidar and I found that, total point count are not the same on the way we count it.
```c
    // Register a callback to receive data
    lidar->Listen([](auto data) {
      auto lidar_data = boost::static_pointer_cast<csd::LidarMeasurement>(data);
      assert(lidar_data != nullptr);

      /////////////////////////////////////////////////////////////////////////
      ///////////////////////// FIRST METHOD //////////////////////////////////
      /////////////////////////////////////////////////////////////////////////
      int totalPoints1 = 0;
      for(unsigned int j = 0; j < lidar_data->GetChannelCount() ; j++){
        totalPoints1 += lidar_data->GetPointCount(j);
      }
      std::cout << "totalPoints1=" << totalPoints1  << std::endl;
      /////////////////////////////////////////////////////////////////////////
      ///////////////////////// SECOND METHOD //////////////////////////////////
      /////////////////////////////////////////////////////////////////////////    
      int totalPoints2 = 0;
      auto iter = lidar_data->begin();
      while (iter != lidar_data->end()){
          totalPoints2++;
          iter++;
      }
      std::cout << "totalPoints2=" << totalPoints2  << std::endl;

      // Just for debug
      exit(0);
    });
````

```sh
Client API version : 0.9.11
Server API version : 0.9.11
totalPoints1=12232
totalPoints2=12221
```

Can you please tell me, which one is correct way to get total points?
Thank you!


Discussion / hints:
@DSantosO could you follow up on this please?
Hello @lapd-viosoft,
Thank you for your report.
We have reproduced the issue and we are looking into it. 



Fix patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 1cff44a6c9a..8707f474699 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -15,6 +15,7 @@
   * CARLA is compatible with the last RoadRunner nomenclature for road assets
   * Fixed a bug when importing a FBX map with some **_** in the FBX name
   * Fix bug in carla.Transform.get_up_vector()
+  * Fix bug in lidar channel point count
   * When setting a global plan at the LocalPlanner, it is now optional to stop the automatic fill of the waypoint buffer
   * API extensions:
   - Added `set_wheel_steer_direction()` function to change the bone angle of each wheel of a vehicle
diff --git a/LibCarla/source/carla/sensor/data/LidarData.h b/LibCarla/source/carla/sensor/data/LidarData.h
index 73d325047d4..cc5e3a89f43 100644
--- a/LibCarla/source/carla/sensor/data/LidarData.h
+++ b/LibCarla/source/carla/sensor/data/LidarData.h
@@ -79,13 +79,10 @@ namespace data {
 
     ~LidarData() = default;
 
-    virtual void ResetSerPoints(std::vector<uint32_t> points_per_channel) {
+    virtual void ResetMemory(std::vector<uint32_t> points_per_channel) {
       DEBUG_ASSERT(GetChannelCount() > points_per_channel.size());
       std::memset(_header.data() + Index::SIZE, 0, sizeof(uint32_t) * GetChannelCount());
 
-      for (auto idxChannel = 0u; idxChannel < GetChannelCount(); ++idxChannel)
-        _header[Index::SIZE + idxChannel] = points_per_channel[idxChannel];
-
       uint32_t total_points = static_cast<uint32_t>(
           std::accumulate(points_per_channel.begin(), points_per_channel.end(), 0));
 
diff --git a/LibCarla/source/carla/sensor/data/SemanticLidarData.h b/LibCarla/source/carla/sensor/data/SemanticLidarData.h
index 30d711d050c..ef0e6f0a61f 100644
--- a/LibCarla/source/carla/sensor/data/SemanticLidarData.h
+++ b/LibCarla/source/carla/sensor/data/SemanticLidarData.h
@@ -110,13 +110,10 @@ namespace data {
       return _header[Index::ChannelCount];
     }
 
-    virtual void ResetSerPoints(std::vector<uint32_t> points_per_channel) {
+    virtual void ResetMemory(std::vector<uint32_t> points_per_channel) {
       DEBUG_ASSERT(GetChannelCount() > points_per_channel.size());
       std::memset(_header.data() + Index::SIZE, 0, sizeof(uint32_t) * GetChannelCount());
 
-      for (auto idxChannel = 0u; idxChannel < GetChannelCount(); ++idxChannel)
-        _header[Index::SIZE + idxChannel] = points_per_channel[idxChannel];
-
       uint32_t total_points = static_cast<uint32_t>(
           std::accumulate(points_per_channel.begin(), points_per_channel.end(), 0));
 
@@ -124,6 +121,11 @@ namespace data {
       _ser_points.reserve(total_points);
     }
 
+    virtual void WriteChannelCount(std::vector<uint32_t> points_per_channel) {
+      for (auto idxChannel = 0u; idxChannel < GetChannelCount(); ++idxChannel)
+        _header[Index::SIZE + idxChannel] = points_per_channel[idxChannel];
+    }
+
     virtual void WritePointSync(SemanticLidarDetection &detection) {
       _ser_points.emplace_back(detection);
     }
diff --git a/Unreal/CarlaUE4/Plugins/Carla/Source/Carla/Sensor/RayCastLidar.cpp b/Unreal/CarlaUE4/Plugins/Carla/Source/Carla/Sensor/RayCastLidar.cpp
index e0214dc01d8..beac45e3ecd 100644
--- a/Unreal/CarlaUE4/Plugins/Carla/Source/Carla/Sensor/RayCastLidar.cpp
+++ b/Unreal/CarlaUE4/Plugins/Carla/Source/Carla/Sensor/RayCastLidar.cpp
@@ -125,13 +125,18 @@ ARayCastLidar::FDetection ARayCastLidar::ComputeDetection(const FHitResult& HitI
   void ARayCastLidar::ComputeAndSaveDetections(const FTransform& SensorTransform) {
     for (auto idxChannel = 0u; idxChannel < Description.Channels; ++idxChannel)
       PointsPerChannel[idxChannel] = RecordedHits[idxChannel].size();
-    LidarData.ResetSerPoints(PointsPerChannel);
+
+    LidarData.ResetMemory(PointsPerChannel);
 
     for (auto idxChannel = 0u; idxChannel < Description.Channels; ++idxChannel) {
       for (auto& hit : RecordedHits[idxChannel]) {
         FDetection Detection = ComputeDetection(hit, SensorTransform);
         if (PostprocessDetection(Detection))
           LidarData.WritePointSync(Detection);
+        else
+          PointsPerChannel[idxChannel]--;
       }
     }
+
+    LidarData.WriteChannelCount(PointsPerChannel);
   }
diff --git a/Unreal/CarlaUE4/Plugins/Carla/Source/Carla/Sensor/RayCastSemanticLidar.cpp b/Unreal/CarlaUE4/Plugins/Carla/Source/Carla/Sensor/RayCastSemanticLidar.cpp
index 7720c335771..7f78f67c2ed 100644
--- a/Unreal/CarlaUE4/Plugins/Carla/Source/Carla/Sensor/RayCastSemanticLidar.cpp
+++ b/Unreal/CarlaUE4/Plugins/Carla/Source/Carla/Sensor/RayCastSemanticLidar.cpp
@@ -160,7 +160,7 @@ void ARayCastSemanticLidar::WritePointAsync(uint32_t channel, FHitResult &detect
 void ARayCastSemanticLidar::ComputeAndSaveDetections(const FTransform& SensorTransform) {
   for (auto idxChannel = 0u; idxChannel < Description.Channels; ++idxChannel)
     PointsPerChannel[idxChannel] = RecordedHits[idxChannel].size();
-  SemanticLidarData.ResetSerPoints(PointsPerChannel);
+  SemanticLidarData.ResetMemory(PointsPerChannel);
 
   for (auto idxChannel = 0u; idxChannel < Description.Channels; ++idxChannel) {
     for (auto& hit : RecordedHits[idxChannel]) {
@@ -169,6 +169,8 @@ void ARayCastSemanticLidar::ComputeAndSaveDetections(const FTransform& SensorTra
       SemanticLidarData.WritePointSync(detection);
     }
   }
+
+  SemanticLidarData.WriteChannelCount(PointsPerChannel);
 }
 
 void ARayCastSemanticLidar::ComputeRawDetection(const FHitResult& HitInfo, const FTransform& SensorTransf, FSemanticDetection& Detection) const
diff --git a/Unreal/CarlaUE4/Plugins/Carla/Source/Carla/Sensor/RayCastSemanticLidar.h b/Unreal/CarlaUE4/Plugins/Carla/Source/Carla/Sensor/RayCastSemanticLidar.h
index 7269c8297a6..51da5495198 100644
--- a/Unreal/CarlaUE4/Plugins/Carla/Source/Carla/Sensor/RayCastSemanticLidar.h
+++ b/Unreal/CarlaUE4/Plugins/Carla/Source/Carla/Sensor/RayCastSemanticLidar.h
@@ -46,7 +46,7 @@ class CARLA_API ARayCastSemanticLidar : public ASensor
   void CreateLasers();
 
   /// Updates LidarMeasurement with the points read in DeltaTime.
-  void SimulateLidar(float DeltaTime);
+  void SimulateLidar(const float DeltaTime);
 
   /// Shoot a laser ray-trace, return whether the laser hit something.
   bool ShootLaser(const float VerticalAngle, float HorizontalAngle, FHitResult &HitResult) const;


Test patch:
diff --git a/PythonAPI/test/smoke/__init__.py b/PythonAPI/test/smoke/__init__.py
index de115f03bdd..b79339f546c 100644
--- a/PythonAPI/test/smoke/__init__.py
+++ b/PythonAPI/test/smoke/__init__.py
@@ -28,16 +28,17 @@ def setUp(self):
         self.testing_address = TESTING_ADDRESS
         self.client = carla.Client(*TESTING_ADDRESS)
         self.client.set_timeout(120.0)
+        self.world = self.client.get_world()
 
     def tearDown(self):
         self.client.load_world("Town03")
+        self.world = None
         self.client = None
 
 
 class SyncSmokeTest(SmokeTest):
     def setUp(self):
         super(SyncSmokeTest, self).setUp()
-        self.world = self.client.get_world()
         self.settings = self.world.get_settings()
         settings = carla.WorldSettings(
             no_rendering_mode=False,
@@ -50,5 +51,4 @@ def tearDown(self):
         self.world.apply_settings(self.settings)
         self.world.tick()
         self.settings = None
-        self.world = None
         super(SyncSmokeTest, self).tearDown()
diff --git a/PythonAPI/test/smoke/test_lidar.py b/PythonAPI/test/smoke/test_lidar.py
new file mode 100644
index 00000000000..1b7255a5f84
--- /dev/null
+++ b/PythonAPI/test/smoke/test_lidar.py
@@ -0,0 +1,176 @@
+# Copyright (c) 2021 Computer Vision Center (CVC) at the Universitat Autonoma de
+# Barcelona (UAB).
+#
+# This work is licensed under the terms of the MIT license.
+# For a copy, see <https://opensource.org/licenses/MIT>.
+
+from . import SyncSmokeTest
+from . import SmokeTest
+
+import carla
+import time
+import math
+import numpy as np
+from enum import Enum
+
+class SensorType(Enum):
+    LIDAR = 1
+    SEMLIDAR = 2
+
+class Sensor():
+    def __init__(self, test, sensor_type, attributes):
+        self.test = test
+        self.world = test.world
+        self.sensor_type = sensor_type
+        self.error = None
+
+        if self.sensor_type == SensorType.LIDAR:
+            self.bp_sensor = self.world.get_blueprint_library().filter("sensor.lidar.ray_cast")[0]
+        elif self.sensor_type == SensorType.SEMLIDAR:
+            self.bp_sensor = self.world.get_blueprint_library().filter("sensor.lidar.ray_cast_semantic")[0]
+        else:
+            self.error = "Unknown type of sensor"
+
+        for key in attributes:
+            self.bp_sensor.set_attribute(key, attributes[key])
+
+        tranf = self.world.get_map().get_spawn_points()[0]
+        self.sensor = self.world.spawn_actor(self.bp_sensor, tranf)
+        self.sensor.listen(lambda sensor_data: self.callback(sensor_data))
+
+    def destroy(self):
+        self.sensor.destroy()
+
+    def callback(self, sensor_data):
+        # Compute the total sum of points adding all channels
+        total_channel_points = 0
+        for i in range(0, sensor_data.channels):
+            total_channel_points += sensor_data.get_point_count(i)
+
+        # Total points iterating in the LidarMeasurement
+        total_detect_points = 0
+        for _detection in sensor_data:
+            total_detect_points += 1
+
+        # Point cloud used with numpy from the raw data
+        if self.sensor_type == SensorType.LIDAR:
+            points = np.frombuffer(sensor_data.raw_data, dtype=np.dtype('f4'))
+            points = np.reshape(points, (int(points.shape[0] / 4), 4))
+            total_np_points = points.shape[0]
+        elif self.sensor_type == SensorType.SEMLIDAR:
+            data = np.frombuffer(sensor_data.raw_data, dtype=np.dtype([
+                ('x', np.float32), ('y', np.float32), ('z', np.float32),
+                ('CosAngle', np.float32), ('ObjIdx', np.uint32), ('ObjTag', np.uint32)]))
+            points = np.array([data['x'], data['y'], data['z']]).T
+            total_np_points = points.shape[0]
+        else:
+            self.error = "It should never reach this point"
+            return
+
+        if total_np_points != total_detect_points:
+            self.error = "The number of points of the raw data does not match with the LidarMeasurament array"
+
+        if total_channel_points != total_detect_points:
+            self.error = "The sum of the points of all channels does not match with the LidarMeasurament array"
+
+    def is_correct(self):
+        return self.error is None
+
+class TestSyncLidar(SyncSmokeTest):
+    def test_lidar_point_count(self):
+        print("TestSyncLidar.test_lidar_point_count")
+        sensors = []
+
+        att_l00={'channels' : '64', 'dropoff_intensity_limit': '0.0', 'dropoff_general_rate': '0.0',
+          'range' : '50', 'points_per_second': '100000', 'rotation_frequency': '20'}
+        att_l01={'channels' : '64', 'range' : '200', 'points_per_second': '500000',
+          'rotation_frequency': '5'}
+        att_l02={'channels' : '64', 'dropoff_intensity_limit': '1.0', 'dropoff_general_rate': '0.0',
+          'range' : '50', 'points_per_second': '100000', 'rotation_frequency': '50'}
+
+        sensors.append(Sensor(self, SensorType.LIDAR, att_l00))
+        sensors.append(Sensor(self, SensorType.LIDAR, att_l01))
+        sensors.append(Sensor(self, SensorType.LIDAR, att_l02))
+
+        for _ in range(0, 10):
+            self.world.tick()
+        time.sleep(0.5)
+
+        for sensor in sensors:
+            sensor.destroy()
+
+        for sensor in sensors:
+            if not sensor.is_correct():
+                self.fail(sensor.error)
+
+
+    def test_semlidar_point_count(self):
+        print("TestSyncLidar.test_semlidar_point_count")
+        sensors = []
+
+        att_s00 = {'channels' : '64', 'range' : '100', 'points_per_second': '100000',
+          'rotation_frequency': '20'}
+        att_s01 = {'channels' : '32', 'range' : '200', 'points_per_second': '500000',
+          'rotation_frequency': '50'}
+
+        sensors.append(Sensor(self, SensorType.SEMLIDAR, att_s00))
+        sensors.append(Sensor(self, SensorType.SEMLIDAR, att_s01))
+
+        for _ in range(0, 10):
+            self.world.tick()
+        time.sleep(0.5)
+
+        for sensor in sensors:
+            sensor.destroy()
+
+        for sensor in sensors:
+            if not sensor.is_correct():
+                self.fail(sensor.error)
+
+
+class TestASyncLidar(SmokeTest):
+    def test_lidar_point_count(self):
+        print("TestASyncLidar.test_lidar_point_count")
+        sensors = []
+
+        att_l00={'channels' : '64', 'dropoff_intensity_limit': '0.0', 'dropoff_general_rate': '0.0',
+          'range' : '50', 'points_per_second': '100000', 'rotation_frequency': '20'}
+        att_l01={'channels' : '64', 'range' : '200', 'points_per_second': '500000',
+          'rotation_frequency': '5'}
+        att_l02={'channels' : '64', 'dropoff_intensity_limit': '1.0', 'dropoff_general_rate': '0.0',
+          'range' : '50', 'points_per_second': '100000', 'rotation_frequency': '50'}
+
+        sensors.append(Sensor(self, SensorType.LIDAR, att_l00))
+        sensors.append(Sensor(self, SensorType.LIDAR, att_l01))
+        sensors.append(Sensor(self, SensorType.LIDAR, att_l02))
+
+        time.sleep(3.0)
+
+        for sensor in sensors:
+            sensor.destroy()
+
+        for sensor in sensors:
+            if not sensor.is_correct():
+                self.fail(sensor.error)
+
+
+    def test_semlidar_point_count(self):
+        print("TestASyncLidar.test_semlidar_point_count")
+        sensors = []
+
+        att_s00 = {'channels' : '64', 'range' : '100', 'points_per_second': '100000',
+          'rotation_frequency': '20'}
+        att_s01 = {'channels' : '32', 'range' : '200', 'points_per_second': '500000',
+          'rotation_frequency': '50'}
+
+        sensors.append(Sensor(self, SensorType.SEMLIDAR, att_s00))
+        sensors.append(Sensor(self, SensorType.SEMLIDAR, att_s01))
+
+        time.sleep(3.0)
+
+        for sensor in sensors:
+            sensor.destroy()
+
+        for sensor in sensors:
+            if not sensor.is_correct():
+                self.fail(sensor.error)
diff --git a/PythonAPI/test/smoke_test_list.txt b/PythonAPI/test/smoke_test_list.txt
index d89c104e47f..0392f7e772a 100644
--- a/PythonAPI/test/smoke_test_list.txt
+++ b/PythonAPI/test/smoke_test_list.txt
@@ -1,1 +1,1 @@
-smoke.test_client smoke.test_sync smoke.test_sensor_determinism smoke.test_props_loading smoke.test_sensor_tick_time smoke.test_map smoke.test_snapshot smoke.test_streamming smoke.test_spawnpoints smoke.test_blueprint smoke.test_world smoke.test_determinism
+smoke.test_client smoke.test_sync smoke.test_sensor_determinism smoke.test_props_loading smoke.test_sensor_tick_time smoke.test_map smoke.test_snapshot smoke.test_lidar smoke.test_streamming smoke.test_spawnpoints smoke.test_blueprint smoke.test_world smoke.test_determinism