An apparatus for decoding an image, the apparatus comprising: an entropy decoder configured to restore quantized residual coefficients and intra prediction information; a prediction mode decoder configured to restore an intra prediction mode on the basis of the intra prediction information; a residual signal decoder configured to restore a residual block by inversely scanning the quantized residual coefficients; a reference pixel generator configured to generate reference pixels corresponding to unavailable reference pixels and to adaptively filter the reference pixels according to the intra prediction mode; a prediction block generator configured to generate a prediction block using the reference pixels determined by the intra prediction mode; and an image reconstructor configured to generate a reconstructed image using the prediction block and the residual block, wherein the reference pixel generator adaptively filters the reference pixels according to a size of the prediction block for intra prediction modes existing between a horizontal mode and an intra prediction mode having a direction of 45° with respect to the horizontal mode, wherein, when the reference pixels are filtered for a second directional intra prediction mode, the reference pixels are also filtered for a first directional intra prediction mode that is closer to the intra prediction mode having the direction of 45° with respect to the horizontal mode than the second directional intra prediction mode is, wherein the second and the first directional intra prediction modes exist between the horizontal mode and the intra prediction mode having the direction of 45° with respect to the horizontal mode, wherein the number of intra prediction modes in which the filter is applied increases as the size of the prediction block increases, wherein the filter is applied to the intra prediction mode having a direction of 45° with respect to the horizontal mode and is not applied to the horizontal mode and the vertical mode, wherein, when it is determined that a size of a transform unit is larger than a predetermined size, the residual signal decoder inversely scans the quantized residual coefficients in the unit of sub-block, wherein the predetermined size is 4x4 or 8x8, wherein the plurality of sub-blocks and the coefficients of each sub-block are inversely scanned according to a scan pattern, wherein the scan pattern for inversely scanning the plurality of sub-blocks and for inversely scanning the coefficients of each sub-block is determined by the intra prediction mode of the prediction block. an entropy decoder configured to restore quantized residual coefficients and intra prediction information; a prediction mode decoder configured to restore an intra prediction mode on the basis of the intra prediction information; a residual signal decoder configured to restore a residual block by inversely scanning the quantized residual coefficients; a reference pixel generator configured to generate reference pixels corresponding to unavailable reference pixels and to adaptively filter the reference pixels according to the intra prediction mode; a prediction block generator configured to generate a prediction block using the reference pixels determined by the intra prediction mode; and an image reconstructor configured to generate a reconstructed image using the prediction block and the residual block, wherein the reference pixel generator adaptively filters the reference pixels according to a size of the prediction block for intra prediction modes existing between a horizontal mode and an intra prediction mode having a direction of 45° with respect to the horizontal mode, wherein, when the reference pixels are filtered for a second directional intra prediction mode, the reference pixels are also filtered for a first directional intra prediction mode that is closer to the intra prediction mode having the direction of 45° with respect to the horizontal mode than the second directional intra prediction mode is, wherein the second and the first directional intra prediction modes exist between the horizontal mode and the intra prediction mode having the direction of 45° with respect to the horizontal mode, wherein the number of intra prediction modes in which the filter is applied increases as the size of the prediction block increases, wherein the filter is applied to the intra prediction mode having a direction of 45° with respect to the horizontal mode and is not applied to the horizontal mode and the vertical mode, wherein, when it is determined that a size of a transform unit is larger than a predetermined size, the residual signal decoder inversely scans the quantized residual coefficients in the unit of sub-block, wherein the predetermined size is 4x4 or 8x8, wherein the plurality of sub-blocks and the coefficients of each sub-block are inversely scanned according to a scan pattern, wherein the scan pattern for inversely scanning the plurality of sub-blocks and for inversely scanning the coefficients of each sub-block is determined by the intra prediction mode of the prediction block.