A computer-implemented method for operating an autonomous driving vehicle, the method comprising: detecting a pedestrian in a plurality of frames of point clouds generated by a LIDAR device installed at an autonomous driving vehicle, ADV; in each of at least two of the plurality of frames of point clouds, generating a minimum bounding box enclosing points (502, 504, 506, 508) corresponding to the pedestrian and excluding points corresponding to limbs of the pedestrian; estimating a moving speed of the pedestrian based at least in part on the minimum bounding boxes across the at least two of the plurality of frames of point clouds; planning a trajectory for the ADV based at least on the moving speed of the pedestrian; and generating control signals to drive the ADV based on the planned trajectory; wherein generating the minimum bounding box further comprises: filtering the points corresponding to the pedestrian to remove lower points and retain only higher points (502, 504, 506), the lower points corresponding to legs and feet of the pedestrian; projecting the higher points (502, 504, 506) in the points corresponding to the pedestrian to a horizontal plane; determining a principal axis associated with the projected higher points on the horizontal plane; projecting the projected higher points (502, 504, 506) on the horizontal plane to the principal axis; and filtering the twice-projected higher points (502, 504, 506) to remove most outlying points (502, 506) in both directions of the principal axis, the most outlying points (502, 506) corresponding to arms and hands of the pedestrian. detecting a pedestrian in a plurality of frames of point clouds generated by a LIDAR device installed at an autonomous driving vehicle, ADV; in each of at least two of the plurality of frames of point clouds, generating a minimum bounding box enclosing points (502, 504, 506, 508) corresponding to the pedestrian and excluding points corresponding to limbs of the pedestrian; estimating a moving speed of the pedestrian based at least in part on the minimum bounding boxes across the at least two of the plurality of frames of point clouds; planning a trajectory for the ADV based at least on the moving speed of the pedestrian; and generating control signals to drive the ADV based on the planned trajectory; wherein generating the minimum bounding box further comprises: filtering the points corresponding to the pedestrian to remove lower points and retain only higher points (502, 504, 506), the lower points corresponding to legs and feet of the pedestrian; projecting the higher points (502, 504, 506) in the points corresponding to the pedestrian to a horizontal plane; determining a principal axis associated with the projected higher points on the horizontal plane; projecting the projected higher points (502, 504, 506) on the horizontal plane to the principal axis; and filtering the twice-projected higher points (502, 504, 506) to remove most outlying points (502, 506) in both directions of the principal axis, the most outlying points (502, 506) corresponding to arms and hands of the pedestrian. filtering the points corresponding to the pedestrian to remove lower points and retain only higher points (502, 504, 506), the lower points corresponding to legs and feet of the pedestrian; projecting the higher points (502, 504, 506) in the points corresponding to the pedestrian to a horizontal plane; determining a principal axis associated with the projected higher points on the horizontal plane; projecting the projected higher points (502, 504, 506) on the horizontal plane to the principal axis; and filtering the twice-projected higher points (502, 504, 506) to remove most outlying points (502, 506) in both directions of the principal axis, the most outlying points (502, 506) corresponding to arms and hands of the pedestrian.