A processor implemented method, comprising: obtaining, by one or more hardware processors, one or more input data comprising skeletal data, depth data and red, green, and blue, RGB, data pertaining to at least one user at one or more time stamps, wherein said one or more input data comprises three dimensional, 3D, coordinates pertaining to one or more skeleton joints; estimating, using said one or more input data, by said one or more hardware processors, in a first set of frames, one or more body joint locations and a body segment length of one or more body segments connected to corresponding skeleton joints to obtain i) an estimate of one or more body joint locations and ii) an estimate of one or more body segment lengths; iteratively identifying, by said one or more hardware processors, one or more probable correct body joint locations in a bounded neighborhood around said estimate of one or more body joint locations based on said depth data and said RGB data; performing, by said one or more hardware processors, a comparison of i) each body segment length from the estimate of one or more body segment length associated with said one or more probable correct body joint locations and ii) a reference length; identifying, by said one or more hardware processors, at least a subset of said one or more probable correct body joint locations as one or more candidate body joint locations based on said comparison; determining a physical orientation of each body segment pertaining to each of said one or more candidate body joint locations by segmenting one or more 3D coordinates of each body segment based on said depth data and performing an analysis on each segmented 3D coordinate; and identifying, from said one or more candidate body joint locations, a body joint location based on a minimal deviation in direction from said physical orientation of a corresponding body segment along with a feature descriptor of said RGB data and said depth data, wherein said minimal deviation is based on said depth data, and wherein said minimal deviation is based on range of motions performed by said user. obtaining, by one or more hardware processors, one or more input data comprising skeletal data, depth data and red, green, and blue, RGB, data pertaining to at least one user at one or more time stamps, wherein said one or more input data comprises three dimensional, 3D, coordinates pertaining to one or more skeleton joints; estimating, using said one or more input data, by said one or more hardware processors, in a first set of frames, one or more body joint locations and a body segment length of one or more body segments connected to corresponding skeleton joints to obtain i) an estimate of one or more body joint locations and ii) an estimate of one or more body segment lengths; iteratively identifying, by said one or more hardware processors, one or more probable correct body joint locations in a bounded neighborhood around said estimate of one or more body joint locations based on said depth data and said RGB data; performing, by said one or more hardware processors, a comparison of i) each body segment length from the estimate of one or more body segment length associated with said one or more probable correct body joint locations and ii) a reference length; identifying, by said one or more hardware processors, at least a subset of said one or more probable correct body joint locations as one or more candidate body joint locations based on said comparison; determining a physical orientation of each body segment pertaining to each of said one or more candidate body joint locations by segmenting one or more 3D coordinates of each body segment based on said depth data and performing an analysis on each segmented 3D coordinate; and identifying, from said one or more candidate body joint locations, a body joint location based on a minimal deviation in direction from said physical orientation of a corresponding body segment along with a feature descriptor of said RGB data and said depth data, wherein said minimal deviation is based on said depth data, and wherein said minimal deviation is based on range of motions performed by said user.