A system for classifying a moving object during a video surveillance the system comprising: a video capturing device configured to capture consecutive silhouette image frames of a video of the moving object in an operating range of the video capturing device; and a memory that stores instructions; a processor that executes instructions to perform operations comprising: dividing the silhouette image frame into an upper half and a lower half; calculating a resize scale factor using dimensions of the upper half of the silhouette image frame; resizing the silhouette image frame by applying the resizing scale factor that facilitates a resized silhouette image frame; computing an average height of the moving object and an average width of the moving object in the resized silhouette image frame; computing a center of gravity of the moving object in the resized silhouette image frame, using only the upper half of the silhouette image frame; dividing the lower half of the silhouette image into two parts comprising a lower left half and a lower right half, wherein the lower half is divided by using a vertical line passing through the center of gravity; calculating a variance of the center of gravity based on consecutive frames of the silhouette image frame corresponding to change in one of the two parts of the lower half with respect to the center of gravity; comparing the average height of the moving object with the average width of the moving object; comparing the variance of the center of gravity with a predetermined threshold value; and classifying the moving object in the silhouette image frame into a predetermined class based on the comparison of the average height of the moving object with the average width of the moving object, and comparison of the variance of the center of gravity with the predetermined threshold value. a video capturing device configured to capture consecutive silhouette image frames of a video of the moving object in an operating range of the video capturing device; and a memory that stores instructions; a processor that executes instructions to perform operations comprising: dividing the silhouette image frame into an upper half and a lower half; calculating a resize scale factor using dimensions of the upper half of the silhouette image frame; resizing the silhouette image frame by applying the resizing scale factor that facilitates a resized silhouette image frame; computing an average height of the moving object and an average width of the moving object in the resized silhouette image frame; computing a center of gravity of the moving object in the resized silhouette image frame, using only the upper half of the silhouette image frame; dividing the lower half of the silhouette image into two parts comprising a lower left half and a lower right half, wherein the lower half is divided by using a vertical line passing through the center of gravity; calculating a variance of the center of gravity based on consecutive frames of the silhouette image frame corresponding to change in one of the two parts of the lower half with respect to the center of gravity; comparing the average height of the moving object with the average width of the moving object; comparing the variance of the center of gravity with a predetermined threshold value; and classifying the moving object in the silhouette image frame into a predetermined class based on the comparison of the average height of the moving object with the average width of the moving object, and comparison of the variance of the center of gravity with the predetermined threshold value. dividing the silhouette image frame into an upper half and a lower half; calculating a resize scale factor using dimensions of the upper half of the silhouette image frame; resizing the silhouette image frame by applying the resizing scale factor that facilitates a resized silhouette image frame; computing an average height of the moving object and an average width of the moving object in the resized silhouette image frame; computing a center of gravity of the moving object in the resized silhouette image frame, using only the upper half of the silhouette image frame; dividing the lower half of the silhouette image into two parts comprising a lower left half and a lower right half, wherein the lower half is divided by using a vertical line passing through the center of gravity; calculating a variance of the center of gravity based on consecutive frames of the silhouette image frame corresponding to change in one of the two parts of the lower half with respect to the center of gravity; comparing the average height of the moving object with the average width of the moving object; comparing the variance of the center of gravity with a predetermined threshold value; and classifying the moving object in the silhouette image frame into a predetermined class based on the comparison of the average height of the moving object with the average width of the moving object, and comparison of the variance of the center of gravity with the predetermined threshold value.