An image encoding apparatus that encodes an image so as to optimize a rate distortion cost of a transform unit, the apparatus comprising: a division device that divides an image having a size 2N*2N of a coding unit into four divided blocks having a block size N*N; an activity computation device that computes an activity pertaining to each block of the plurality of blocks; and a transform unit size determination device that determines a size of the transform unit within the image having the size 2N*2N of the coding unit, characterized in that the transform unit size determination device: - compares a difference in the computed activities pertaining to every two adjacent blocks among the four divided blocks with a first predetermined threshold, - when the difference in the computed activities between any two adjacent blocks among the four divided blocks is smaller than or equal to the first predetermined threshold, determines the block size N*N as the size of the transform unit, - when the difference in the computed activities between any two adjacent blocks among the four divided blocks is greater than the first predetermined threshold, compares the difference in the computed activities pertaining to every two adjacent blocks among the four divided blocks with a second predetermined threshold, the second predetermined threshold being greater than the first predetermined threshold, and - when the difference in the computed activities between any two adjacent blocks among the four divided blocks is smaller than or equal to the second predetermined threshold, determines a block size N/2*N/2 as the size of the transform unit, otherwise determines a block size N/4*N/4 as the size of the transform unit. a division device that divides an image having a size 2N*2N of a coding unit into four divided blocks having a block size N*N; an activity computation device that computes an activity pertaining to each block of the plurality of blocks; and a transform unit size determination device that determines a size of the transform unit within the image having the size 2N*2N of the coding unit, characterized in that the transform unit size determination device: - compares a difference in the computed activities pertaining to every two adjacent blocks among the four divided blocks with a first predetermined threshold, - when the difference in the computed activities between any two adjacent blocks among the four divided blocks is smaller than or equal to the first predetermined threshold, determines the block size N*N as the size of the transform unit, - when the difference in the computed activities between any two adjacent blocks among the four divided blocks is greater than the first predetermined threshold, compares the difference in the computed activities pertaining to every two adjacent blocks among the four divided blocks with a second predetermined threshold, the second predetermined threshold being greater than the first predetermined threshold, and - when the difference in the computed activities between any two adjacent blocks among the four divided blocks is smaller than or equal to the second predetermined threshold, determines a block size N/2*N/2 as the size of the transform unit, otherwise determines a block size N/4*N/4 as the size of the transform unit. - compares a difference in the computed activities pertaining to every two adjacent blocks among the four divided blocks with a first predetermined threshold, - when the difference in the computed activities between any two adjacent blocks among the four divided blocks is smaller than or equal to the first predetermined threshold, determines the block size N*N as the size of the transform unit, - when the difference in the computed activities between any two adjacent blocks among the four divided blocks is greater than the first predetermined threshold, compares the difference in the computed activities pertaining to every two adjacent blocks among the four divided blocks with a second predetermined threshold, the second predetermined threshold being greater than the first predetermined threshold, and - when the difference in the computed activities between any two adjacent blocks among the four divided blocks is smaller than or equal to the second predetermined threshold, determines a block size N/2*N/2 as the size of the transform unit, otherwise determines a block size N/4*N/4 as the size of the transform unit.