A computer-implemented method comprising: identifying, by one or more computing devices, a reference image; identifying, by the one or more computing devices, a set of potential target images for the reference image, wherein each of the reference and potential target images is associated with: a depth map defining the 3D location of each pixel of the image in real world coordinates, and location and orientation information for a camera that captured the image; determining, by the one or more computing devices, a drag vector for user input relative to the reference image; after the drag vector is determined, for each particular potential target image of the set of potential target images, determining, by the one or more computing devices, an associated cost based at least in part on a cost function and the drag vector, wherein the cost function is determined for a particular motion result, as a weighted sum of one or more cost terms including one or more of a drag flow cost, a rotation cost, an opposite rotation cost, a non-rotation cost, a translation cost, an opposite translation cost, a non-translation cost and an overlap cost, and selecting, by the one or more computing devices, a target image from the set of potential target images based on the determined associated cost, wherein: the motion result comprises at least one of a panning motion, a sliding motion, and an orbiting motion, for the panning motion, weights are assigned for the drag flow cost, the rotation cost and the non-translation cost, for the sliding motion, weights are assigned for the drag flow cost, the non-rotation cost and the translation cost, and for the orbiting motion, weights are assigned for the opposite rotation cost and the translation cost. identifying, by one or more computing devices, a reference image; identifying, by the one or more computing devices, a set of potential target images for the reference image, wherein each of the reference and potential target images is associated with: a depth map defining the 3D location of each pixel of the image in real world coordinates, and location and orientation information for a camera that captured the image; a depth map defining the 3D location of each pixel of the image in real world coordinates, and location and orientation information for a camera that captured the image; determining, by the one or more computing devices, a drag vector for user input relative to the reference image; after the drag vector is determined, for each particular potential target image of the set of potential target images, determining, by the one or more computing devices, an associated cost based at least in part on a cost function and the drag vector, wherein the cost function is determined for a particular motion result, as a weighted sum of one or more cost terms including one or more of a drag flow cost, a rotation cost, an opposite rotation cost, a non-rotation cost, a translation cost, an opposite translation cost, a non-translation cost and an overlap cost, and selecting, by the one or more computing devices, a target image from the set of potential target images based on the determined associated cost, wherein: the motion result comprises at least one of a panning motion, a sliding motion, and an orbiting motion, for the panning motion, weights are assigned for the drag flow cost, the rotation cost and the non-translation cost, for the sliding motion, weights are assigned for the drag flow cost, the non-rotation cost and the translation cost, and for the orbiting motion, weights are assigned for the opposite rotation cost and the translation cost.