A method of processing an image based on an intra-prediction mode, comprising steps of: deriving an intra-prediction mode of a processing block; splitting the processing block into a plurality of rectangular transform units based on the intra-prediction mode of the processing block when the intra-prediction mode has directivity; splitting the processing block into a plurality of transform units based on a square quadtree split method when the intra-prediction mode does not have directivity; constructing a reference samples (901c, 1001c, 1002c) for the split processing block, wherein a number of the reference samples is determined based on a size of the split processing block and a split form of the split processing block; and performing intra-prediction and reconstruction on each of the transform units using the reference samples, wherein the split direction of the split processing block is horizontal in a vertical intraprediction mode, the size of each transform unit is 2NxN/2, the number of the reference samples neighboring the top boundary of each transform unit and the number of the reference samples neighboring the top right of each transform unit is 3N, the number of the reference samples neighboring the left boundary of each transform unit is N/2, and the number of the reference samples neighboring the top left of each transform unit is 1, wherein the split direction of the split processing block is vertical in a horizontal intra-prediction mode, the size of each transform unit is N/2x2N, the number of the reference samples neighboring the left boundary of each transform unit and the number of the reference samples neighboring the bottom left of each transform unit is 3N, the number of the reference samples neighboring the top boundary of each transform unit is N/2, and the number of the reference samples neighboring the top left of each transform unit is 1, wherein the split direction of the split processing block is 45° in a 135° direction prediction mode, the number of the reference samples of each transform unit is determined based on a boundary of each transform unit, and wherein the reference samples of each transform unit include neighboring samples of the boundary of each transform unit. deriving an intra-prediction mode of a processing block; splitting the processing block into a plurality of rectangular transform units based on the intra-prediction mode of the processing block when the intra-prediction mode has directivity; splitting the processing block into a plurality of transform units based on a square quadtree split method when the intra-prediction mode does not have directivity; constructing a reference samples (901c, 1001c, 1002c) for the split processing block, wherein a number of the reference samples is determined based on a size of the split processing block and a split form of the split processing block; and performing intra-prediction and reconstruction on each of the transform units using the reference samples, wherein the split direction of the split processing block is horizontal in a vertical intraprediction mode, the size of each transform unit is 2NxN/2, the number of the reference samples neighboring the top boundary of each transform unit and the number of the reference samples neighboring the top right of each transform unit is 3N, the number of the reference samples neighboring the left boundary of each transform unit is N/2, and the number of the reference samples neighboring the top left of each transform unit is 1, wherein the split direction of the split processing block is vertical in a horizontal intra-prediction mode, the size of each transform unit is N/2x2N, the number of the reference samples neighboring the left boundary of each transform unit and the number of the reference samples neighboring the bottom left of each transform unit is 3N, the number of the reference samples neighboring the top boundary of each transform unit is N/2, and the number of the reference samples neighboring the top left of each transform unit is 1, wherein the split direction of the split processing block is 45° in a 135° direction prediction mode, the number of the reference samples of each transform unit is determined based on a boundary of each transform unit, and wherein the reference samples of each transform unit include neighboring samples of the boundary of each transform unit.