A video coding apparatus in which motion prediction is performed on each picture of an input video signal in units of coding units using a temporal correlation of the picture, and a coding process is performed on a difference between the input video signal and a predicted image generated on the basis of result of the motion prediction, the video coding apparatus comprising: a motion prediction unit configured to perform the motion prediction for each of coding units obtained by dividing a coding tree unit into a plurality of different sizes so as to obtain the first evaluation value for each of the coding units having the different sizes, the coding tree unit being obtained by dividing an original image serving as the picture of the input video signal into a predetermined size, the first evaluation value being a value which is obtained by adding a coding cost to an absolute value sum of differences between pixels of each of the coding units and pixels of an area corresponding to each of the coding units on a reference image; a coding unit size determination unit configured to determine, as a coding unit size, a size of a coding unit used to code a predicted motion vector on the coding tree unit of the original image on the basis of the first evaluation value; a prediction error calculating unit configured to calculate a prediction error representing a difference between a coding unit having the coding unit size determined by the coding unit size determination unit and the predicted image; a merge size determination unit configured to determine, as a merge size, a size of a coding unit used for coding of merge mode on the coding tree unit using the coding unit size determined by the coding unit size determination unit as a reference on the basis of comparison between a threshold value and an indicator obtained on the basis of the prediction error; a merge mode evaluation unit configured to evaluate each of merge candidates on a coding unit of the merge size determined by the merge size determination unit so as to supply a second evaluation value of each of the merge candidates; and an inter-prediction mode determination unit configured to determine a mode used for coding on the basis of the second evaluation value obtained by the merge mode evaluation unit and the first evaluation value of the coding unit size determined by the coding unit size determination unit. a motion prediction unit configured to perform the motion prediction for each of coding units obtained by dividing a coding tree unit into a plurality of different sizes so as to obtain the first evaluation value for each of the coding units having the different sizes, the coding tree unit being obtained by dividing an original image serving as the picture of the input video signal into a predetermined size, the first evaluation value being a value which is obtained by adding a coding cost to an absolute value sum of differences between pixels of each of the coding units and pixels of an area corresponding to each of the coding units on a reference image; a coding unit size determination unit configured to determine, as a coding unit size, a size of a coding unit used to code a predicted motion vector on the coding tree unit of the original image on the basis of the first evaluation value; a prediction error calculating unit configured to calculate a prediction error representing a difference between a coding unit having the coding unit size determined by the coding unit size determination unit and the predicted image; a merge size determination unit configured to determine, as a merge size, a size of a coding unit used for coding of merge mode on the coding tree unit using the coding unit size determined by the coding unit size determination unit as a reference on the basis of comparison between a threshold value and an indicator obtained on the basis of the prediction error; a merge mode evaluation unit configured to evaluate each of merge candidates on a coding unit of the merge size determined by the merge size determination unit so as to supply a second evaluation value of each of the merge candidates; and an inter-prediction mode determination unit configured to determine a mode used for coding on the basis of the second evaluation value obtained by the merge mode evaluation unit and the first evaluation value of the coding unit size determined by the coding unit size determination unit.