An image decoding apparatus for decoding an image on a per coding unit basis, said image having luminance data, first chrominance data, and second chrominance data in either one of 4:2:0 format and 4:2:2 format, said image being divided into a plurality of coding units, each of the coding units being divided into one or more transform units, wherein each coding unit consists of one or more blocks and each of said blocks consists of one or more transform units, the apparatus comprising: a decoding unit configured to decode, for each block, transformed luminance data, first transformed chrominance data, and second transformed chrominance data from a bitstream in which the transformed luminance data, the first transformed chrominance data, and the second transformed chrominance data of the coding unit are grouped on a per block basis; a luminance inverse transforming unit configured to apply an inverse frequency transform to the transformed luminance data of each transform unit; a determining unit configured to determine, for each transform unit, whether a first chroma size of the transform unit is smaller than a predetermined size; a combining unit configured to combine, when the first chroma size of the transform unit is smaller than the predetermined size, the transform units of the corresponding block; a chrominance inverse transforming unit (303, 304) configured to apply the inverse frequency transform to each of the first transformed chrominance data and the second transformed chrominance data of the combined transform units, when the first chroma size of the transform unit is smaller than the predetermined size, and to apply the inverse frequency transform to each of the first transformed chrominance data and the second transformed chrominance data of the transform unit, when the first chroma size of the transform unit is not smaller than the predetermined size. a decoding unit configured to decode, for each block, transformed luminance data, first transformed chrominance data, and second transformed chrominance data from a bitstream in which the transformed luminance data, the first transformed chrominance data, and the second transformed chrominance data of the coding unit are grouped on a per block basis; a luminance inverse transforming unit configured to apply an inverse frequency transform to the transformed luminance data of each transform unit; a determining unit configured to determine, for each transform unit, whether a first chroma size of the transform unit is smaller than a predetermined size; a combining unit configured to combine, when the first chroma size of the transform unit is smaller than the predetermined size, the transform units of the corresponding block; a chrominance inverse transforming unit (303, 304) configured to apply the inverse frequency transform to each of the first transformed chrominance data and the second transformed chrominance data of the combined transform units, when the first chroma size of the transform unit is smaller than the predetermined size, and to apply the inverse frequency transform to each of the first transformed chrominance data and the second transformed chrominance data of the transform unit, when the first chroma size of the transform unit is not smaller than the predetermined size.