A 3D multi view encoding apparatus, comprising: an encoder (101, 104, 108, 114) that encodes an input image sequence and generates and outputs a three dimensional stream including a basic stream and an extended stream including a plurality of groups of pictures, GOPs, wherein the basic stream is a stream obtained by encoding an image sequence retrieved from one eye perspective in a 3D video, and the extended stream is a stream obtained by encoding an image sequence retrieved from the other eye perspective in the 3D video in view of a difference between the one image sequence and the other image sequence; a first virtual buffer (102, 301) that performs a buffer simulation for a current GOP of decoding the basic stream output from the encoder based on information about a specific delay value of a previous GOP; a second virtual buffer (105, 303) that performs a buffer simulation for the current GOP of decoding the three dimensional stream output from the encoder based on information about the specific delay value of the previous GOP; a first calculator (103, 109, 302, 307) that calculates a first delay value as a delay value of the basic stream for the current GOP based on a result of the buffer simulation for the current GOP performed by the first virtual buffer, a second calculator (106, 110, 304, 308) that calculates a second delay value as a delay value of the three dimensional stream for the current GOP based on a result of the buffer simulation for the current GOP performed by the second virtual buffer; and a setter (107, 113, 305, 311) that sets the specific delay value for the current GOP based on the calculated first delay value and second delay value, wherein the setter sets the specific delay value to a value not more than smaller one of the first delay value and the second delay value, the first virtual buffer and the second virtual buffer perform the buffer simulation for current, n+1th GOP based on information about the specific delay value of the previous, nth GOP set by the setter in the basic stream and three dimensional stream, and the encoder (101, 104, 108, 114) writes the specific delay value as a delay value for the basic stream and as a delay value of for the extended stream into the three dimensional stream. an encoder (101, 104, 108, 114) that encodes an input image sequence and generates and outputs a three dimensional stream including a basic stream and an extended stream including a plurality of groups of pictures, GOPs, wherein the basic stream is a stream obtained by encoding an image sequence retrieved from one eye perspective in a 3D video, and the extended stream is a stream obtained by encoding an image sequence retrieved from the other eye perspective in the 3D video in view of a difference between the one image sequence and the other image sequence; a first virtual buffer (102, 301) that performs a buffer simulation for a current GOP of decoding the basic stream output from the encoder based on information about a specific delay value of a previous GOP; a second virtual buffer (105, 303) that performs a buffer simulation for the current GOP of decoding the three dimensional stream output from the encoder based on information about the specific delay value of the previous GOP; a first calculator (103, 109, 302, 307) that calculates a first delay value as a delay value of the basic stream for the current GOP based on a result of the buffer simulation for the current GOP performed by the first virtual buffer, a second calculator (106, 110, 304, 308) that calculates a second delay value as a delay value of the three dimensional stream for the current GOP based on a result of the buffer simulation for the current GOP performed by the second virtual buffer; and a setter (107, 113, 305, 311) that sets the specific delay value for the current GOP based on the calculated first delay value and second delay value, wherein the setter sets the specific delay value to a value not more than smaller one of the first delay value and the second delay value, the first virtual buffer and the second virtual buffer perform the buffer simulation for current, n+1th GOP based on information about the specific delay value of the previous, nth GOP set by the setter in the basic stream and three dimensional stream, and the encoder (101, 104, 108, 114) writes the specific delay value as a delay value for the basic stream and as a delay value of for the extended stream into the three dimensional stream.