A position and orientation measurement device for measuring a position and orientation of an object to be measured, the device comprising: storage means configured to store a three-dimensional shape model of the object; grayscale image input means configured to input a grayscale image of the object; distance image input means configured to input a distance image of the object; and position and orientation calculation means (140, 160) configured to calculate the position and orientation of the object using the grayscale image and the distance image, characterized in that the position and orientation measurement device further comprises: approximate position and orientation input means configured to input an approximate position and orientation of the object with respect to the position and orientation measurement device, wherein the position and orientation calculation means (140, 160) is configured to project line segments of the three-dimensional shape model of the object onto the grayscale image based on the approximate position and orientation of the object, to detect edges that fit the projected line segments from the greyscale image, to obtain a first association between the projected line segments and the detected edges on the grayscale image, to calculate a first position and orientation on the basis of the first association, and to update the approximate position and orientation with the calculated first position and orientation, and thereafter the position and orientation calculation means (140, 160) is configured to project the three-dimensional shape model onto the distance image based on the updated approximate position and orientation of the object, to obtain a second association between elements of the three-dimensional shape model and pixels of the distance image, to calculate a second position and orientation on the basis of the second association, and to update the approximate position and orientation with the calculated second position and orientation. storage means configured to store a three-dimensional shape model of the object; grayscale image input means configured to input a grayscale image of the object; distance image input means configured to input a distance image of the object; and position and orientation calculation means (140, 160) configured to calculate the position and orientation of the object using the grayscale image and the distance image, characterized in that the position and orientation measurement device further comprises: approximate position and orientation input means configured to input an approximate position and orientation of the object with respect to the position and orientation measurement device, wherein the position and orientation calculation means (140, 160) is configured to project line segments of the three-dimensional shape model of the object onto the grayscale image based on the approximate position and orientation of the object, to detect edges that fit the projected line segments from the greyscale image, to obtain a first association between the projected line segments and the detected edges on the grayscale image, to calculate a first position and orientation on the basis of the first association, and to update the approximate position and orientation with the calculated first position and orientation, and thereafter the position and orientation calculation means (140, 160) is configured to project the three-dimensional shape model onto the distance image based on the updated approximate position and orientation of the object, to obtain a second association between elements of the three-dimensional shape model and pixels of the distance image, to calculate a second position and orientation on the basis of the second association, and to update the approximate position and orientation with the calculated second position and orientation. approximate position and orientation input means configured to input an approximate position and orientation of the object with respect to the position and orientation measurement device, wherein the position and orientation calculation means (140, 160) is configured to project line segments of the three-dimensional shape model of the object onto the grayscale image based on the approximate position and orientation of the object, to detect edges that fit the projected line segments from the greyscale image, to obtain a first association between the projected line segments and the detected edges on the grayscale image, to calculate a first position and orientation on the basis of the first association, and to update the approximate position and orientation with the calculated first position and orientation, and thereafter the position and orientation calculation means (140, 160) is configured to project the three-dimensional shape model onto the distance image based on the updated approximate position and orientation of the object, to obtain a second association between elements of the three-dimensional shape model and pixels of the distance image, to calculate a second position and orientation on the basis of the second association, and to update the approximate position and orientation with the calculated second position and orientation.