Question: How is the sequential nature of the story captured?
Answer: we provide the decoder with the context of the whole sequence and the content of the current image (i.e. global and local information) to generate the corresponding text that will contribute to the overall story
Alternative answer: The encoder takes the images in order, one at every timestep $t$ . At time $t=5$ , we obtain the context vector through $h_e^{(t)}$ (represented by $\mathbf {Z}$ ). This vector is used to initialize each decoder's hidden state while the first input to each decoder is its corresponding image embedding $e(I_i)$ . Each decoder generates a sequence of words $\lbrace p_1,...,p_{n}\rbrace $ for each image in the sequence.