An image decoding device for decoding an image in a prediction unit using an inter prediction scheme in which a first reference prediction list and/or a second reference prediction list are used, wherein the first reference prediction list is a list of reference images to be used for P and/or B slice, and the second reference prediction list is a list of reference images to be used for B slice, the image decoding device comprising: a merge motion compensation parameter derivation unit configured to decode a merge index and to derive a first prediction list utilization flag indicating whether or not the first reference prediction list is to be used for inter prediction, and a second prediction list utilization flag indicating whether or not the second reference prediction list is to be used for inter prediction; and a bi-prediction/uni-prediction conversion unit, wherein: if (i) a value of the first prediction list utilization flag is equal to a first value, which indicates that the first reference prediction list is to be used for inter prediction, (ii) a value of the second prediction list utilization flag is equal to the first value, which indicates that the second reference prediction list is to be used for inter prediction, and (iii) the prediction unit has a size of 8x4 or 4x8, the bi-prediction/uni-prediction conversion unit is configured to convert the value of the second prediction list utilization flag to a second value so that the second reference prediction list is not to be used, and else to not convert the value of the second prediction list utilization flag, and wherein: the merge index is an index for a merge candidate, and the merge index is decoded, in a case that a target prediction unit is a skip coding unit or a merge prediction unit. a merge motion compensation parameter derivation unit configured to decode a merge index and to derive a first prediction list utilization flag indicating whether or not the first reference prediction list is to be used for inter prediction, and a second prediction list utilization flag indicating whether or not the second reference prediction list is to be used for inter prediction; and a bi-prediction/uni-prediction conversion unit, wherein: if (i) a value of the first prediction list utilization flag is equal to a first value, which indicates that the first reference prediction list is to be used for inter prediction, (ii) a value of the second prediction list utilization flag is equal to the first value, which indicates that the second reference prediction list is to be used for inter prediction, and (iii) the prediction unit has a size of 8x4 or 4x8, the bi-prediction/uni-prediction conversion unit is configured to convert the value of the second prediction list utilization flag to a second value so that the second reference prediction list is not to be used, and else to not convert the value of the second prediction list utilization flag, and wherein: the merge index is an index for a merge candidate, and the merge index is decoded, in a case that a target prediction unit is a skip coding unit or a merge prediction unit. the merge index is an index for a merge candidate, and the merge index is decoded, in a case that a target prediction unit is a skip coding unit or a merge prediction unit.