A method for providing trajectory information about a video scene, the method comprising: obtaining (810, 815) video scene metadata that includes observations regarding trajectories having spatial location coordinates and additional object features at consecutive time instances for a plurality of objects moving through the video scene; normalizing, using sampling or interpolation, each of the trajectories into a unified length, wherein normalizing trajectories into a unified length includes each trajectory having a same number of observations; clustering each of the plurality of the normalized trajectories into trajectory groups according to relative distances from each other using the spatial location coordinates of the trajectories; extracting statistical features about each trajectory group, wherein the statistical features are extracted by calculating an interpolated value of a statistical feature at a point on another trajectory of the group, by using interpolation between a first value of the statistical feature at a first location along a path of a trajectory of the group and a second value of the statistical feature at a second location along the path, wherein the other trajectory is within an influential region of the trajectory having a boundary having a distance away from a closest point on the trajectory that is a preset portion of an interpolated object width at closest locations; wherein for each spatial location, a group probability is determined for belonging to a group depending on whether the spatial location falls within the influential region of one or more constituent trajectories of a group; wherein statistical object features for each group are determined for each spatial location using the object features of all constituent trajectories from that group, for which the spatial location falls within the influential region of a constituent trajectory; classifying each trajectory in the plurality of trajectories as a normal trajectory or an abnormal trajectory based at least in part on the clustering and the statistical object features of each cluster; extracting semantic data about each trajectory group or each normal trajectory, wherein semantic features are exclusively limited to be based on features of trajectories within a cluster which have a coherent moving direction; and annotating each trajectory with the semantic data. obtaining (810, 815) video scene metadata that includes observations regarding trajectories having spatial location coordinates and additional object features at consecutive time instances for a plurality of objects moving through the video scene; normalizing, using sampling or interpolation, each of the trajectories into a unified length, wherein normalizing trajectories into a unified length includes each trajectory having a same number of observations; clustering each of the plurality of the normalized trajectories into trajectory groups according to relative distances from each other using the spatial location coordinates of the trajectories; extracting statistical features about each trajectory group, wherein the statistical features are extracted by calculating an interpolated value of a statistical feature at a point on another trajectory of the group, by using interpolation between a first value of the statistical feature at a first location along a path of a trajectory of the group and a second value of the statistical feature at a second location along the path, wherein the other trajectory is within an influential region of the trajectory having a boundary having a distance away from a closest point on the trajectory that is a preset portion of an interpolated object width at closest locations; wherein for each spatial location, a group probability is determined for belonging to a group depending on whether the spatial location falls within the influential region of one or more constituent trajectories of a group; wherein statistical object features for each group are determined for each spatial location using the object features of all constituent trajectories from that group, for which the spatial location falls within the influential region of a constituent trajectory; classifying each trajectory in the plurality of trajectories as a normal trajectory or an abnormal trajectory based at least in part on the clustering and the statistical object features of each cluster; extracting semantic data about each trajectory group or each normal trajectory, wherein semantic features are exclusively limited to be based on features of trajectories within a cluster which have a coherent moving direction; and annotating each trajectory with the semantic data.