An apparatus for tracking and reconstructing three-dimensional objects, comprising: a data processing unit, adapted to process image data provided by an image acquisition unit, said image data pertaining to a scene containing a three-dimensional object, wherein said image data comprises image depth data; wherein said data processing unit is adapted to compute a three-dimensional reconstruction of said object from said image data in terms of a function that represents a surface of said object, wherein said function representing said surface is chosen such that said surface of said object corresponds to an equipotential surface of said function; and wherein said data processing unit is adapted to determine a relative position of said image acquisition unit with respect to said object directly from fitting said image depth data provided by said image acquisition unit to said function representing said surface; wherein said data processing unit is adapted to iteratively update said function representing said surface based on an observed relative position of said image acquisition unit with respect to said object; wherein said data processing unit is adapted to update said function by maximizing an observation likelihood given in terms of a difference between a distance of said image acquisition unit from said surface determined from said relative position and a corresponding distance between said image acquisition unit and said surface, said corresponding distance being observed by means of said image acquisition unit, said maximization being conducted for said function; wherein said difference is a weighted difference; characterized in that said weights are Gaussian weights, wherein said weights are chosen such that they have a minimum value larger than zero, irrespective of a value of said difference. a data processing unit, adapted to process image data provided by an image acquisition unit, said image data pertaining to a scene containing a three-dimensional object, wherein said image data comprises image depth data; wherein said data processing unit is adapted to compute a three-dimensional reconstruction of said object from said image data in terms of a function that represents a surface of said object, wherein said function representing said surface is chosen such that said surface of said object corresponds to an equipotential surface of said function; and wherein said data processing unit is adapted to determine a relative position of said image acquisition unit with respect to said object directly from fitting said image depth data provided by said image acquisition unit to said function representing said surface; wherein said data processing unit is adapted to iteratively update said function representing said surface based on an observed relative position of said image acquisition unit with respect to said object; wherein said data processing unit is adapted to update said function by maximizing an observation likelihood given in terms of a difference between a distance of said image acquisition unit from said surface determined from said relative position and a corresponding distance between said image acquisition unit and said surface, said corresponding distance being observed by means of said image acquisition unit, said maximization being conducted for said function; wherein said difference is a weighted difference; characterized in that said weights are Gaussian weights, wherein said weights are chosen such that they have a minimum value larger than zero, irrespective of a value of said difference.