A method of estimating a motion of a panorama image containing 360° omni-directional view information along an x-axis, the method comprising: padding a padding region connected to the left side of a basic reference frame to be used for motion estimation of the panorama image using a right border region of the basic reference frame, padding a padding region connected to the right side of the basic reference frame using a left border region thereof, and making a reference frame by using the padded basic reference frame as the reference frame; estimating a motion vector of a current data unit of the panorama image, using motion vectors of a plurality of previous data units adjacent to the current data unit; determining a value of each pixel of a reference data unit indicated by the estimated motion vector from the reference frame (i) as the value of the pixel when the pixel belongs to the reference frame; and (ii) as the value of another pixel with an x coordinate determined by setting an x-coordinate of the pixel to a value obtained by adding a distance on the x-axis between a border of the basic reference frame adjacent to the pixel and the pixel to an x-coordinate of the opposite border of the basic reference frame when the pixel is located outside the right border of the reference frame, and (iii) as the value of another pixel with an x coordinate determined by setting an x-coordinate of the pixel to a value obtained by subtracting a distance on the x-axis between the border of the basic reference frame adjacent to the pixel and the pixel from the x-coordinate of the opposite border of the basic reference frame when the pixel is located outside the left border of the reference frame; and determining a similarity between the current data unit and the reference data unit using a predetermined evaluation function, wherein the motion vectors of the plurality of previous data units are detected before the motion vector of the current data unit is estimated. padding a padding region connected to the left side of a basic reference frame to be used for motion estimation of the panorama image using a right border region of the basic reference frame, padding a padding region connected to the right side of the basic reference frame using a left border region thereof, and making a reference frame by using the padded basic reference frame as the reference frame; estimating a motion vector of a current data unit of the panorama image, using motion vectors of a plurality of previous data units adjacent to the current data unit; determining a value of each pixel of a reference data unit indicated by the estimated motion vector from the reference frame (i) as the value of the pixel when the pixel belongs to the reference frame; and (ii) as the value of another pixel with an x coordinate determined by setting an x-coordinate of the pixel to a value obtained by adding a distance on the x-axis between a border of the basic reference frame adjacent to the pixel and the pixel to an x-coordinate of the opposite border of the basic reference frame when the pixel is located outside the right border of the reference frame, and (iii) as the value of another pixel with an x coordinate determined by setting an x-coordinate of the pixel to a value obtained by subtracting a distance on the x-axis between the border of the basic reference frame adjacent to the pixel and the pixel from the x-coordinate of the opposite border of the basic reference frame when the pixel is located outside the left border of the reference frame; and determining a similarity between the current data unit and the reference data unit using a predetermined evaluation function, wherein the motion vectors of the plurality of previous data units are detected before the motion vector of the current data unit is estimated.