A method of restoring a resolution of an image, the method comprising: up-scaling, by a processor, a criterion image (201; 401; 901, 908; 1001, 1003) based on a resolution of a reference image (203; 402; 903, 910; 1002) neighboring the criterion image; generating detailed information, based on the reference image; and restoring a resolution of the criterion image using the up-scaled criterion image (202; 902, 909; 1004, 1005) and the detailed information, wherein the restoring comprises restoring a high frequency element of the criterion image, and characterized by said generating comprising: down-scaling the reference image based on the resolution of the criterion image; up-scaling the down-scaled reference image (204; 905, 911; 1007) based on the resolution of the reference image; and generating the detailed information using the up-scaled reference image (205; 906, 912; 1008) and the reference image; said restoring a resolution of the criterion image using the up-scaled criterion image comprising compensating for a viewpoint difference between the criterion image and the reference image; wherein the criterion image and the reference image are parts of a multi-view image and correspond to a low resolution image and a high resolution image, respectively, the reference image having a viewpoint different from a viewpoint of the criterion image; wherein the detailed information comprises filter coefficient information to restore a difference between the up-scaled reference image and the reference image, or difference image information generated by removing the up-scaled reference image from the reference image, said restoring comprising filtering the up-scaled criterion image based on the filter coefficients or adding the difference image information and the up-scaled criterion image, respectively. up-scaling, by a processor, a criterion image (201; 401; 901, 908; 1001, 1003) based on a resolution of a reference image (203; 402; 903, 910; 1002) neighboring the criterion image; generating detailed information, based on the reference image; and restoring a resolution of the criterion image using the up-scaled criterion image (202; 902, 909; 1004, 1005) and the detailed information, wherein the restoring comprises restoring a high frequency element of the criterion image, and characterized by said generating comprising: down-scaling the reference image based on the resolution of the criterion image; up-scaling the down-scaled reference image (204; 905, 911; 1007) based on the resolution of the reference image; and generating the detailed information using the up-scaled reference image (205; 906, 912; 1008) and the reference image; down-scaling the reference image based on the resolution of the criterion image; up-scaling the down-scaled reference image (204; 905, 911; 1007) based on the resolution of the reference image; and generating the detailed information using the up-scaled reference image (205; 906, 912; 1008) and the reference image; said restoring a resolution of the criterion image using the up-scaled criterion image comprising compensating for a viewpoint difference between the criterion image and the reference image; wherein the criterion image and the reference image are parts of a multi-view image and correspond to a low resolution image and a high resolution image, respectively, the reference image having a viewpoint different from a viewpoint of the criterion image; wherein the detailed information comprises filter coefficient information to restore a difference between the up-scaled reference image and the reference image, or difference image information generated by removing the up-scaled reference image from the reference image, said restoring comprising filtering the up-scaled criterion image based on the filter coefficients or adding the difference image information and the up-scaled criterion image, respectively.