Image Generation with LSTM

username: Thabang_Lukhetho
cooked: <p>I want to use a LSTM to generate images, I have inputs images of (30,2,32,32) and target images of (30,1,32,32), How should I structure my data such I will be able generate images using the following architecture ? Or is it even possible to use this simple architecture to get reasonable results?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/5/25d2cdbf5dedb3ed14fefee376d37eff8c4d4a05.png" data-download-href="https://discuss.pytorch.org/uploads/default/25d2cdbf5dedb3ed14fefee376d37eff8c4d4a05" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/5/25d2cdbf5dedb3ed14fefee376d37eff8c4d4a05.png" alt="image" data-base62-sha1="5oBjisskJ1xnOBk4HKLoJpKHepn" width="443" height="500" data-dominant-color="232424"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">810×913 46.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: JakeAndFinn
cooked: <p>To generate images I would use a GAN over a LSTM.<br>
Take a look at these GANs for image Generation.<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/MatthewR2D2/Pytorch/tree/master/GANS" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars0.githubusercontent.com/u/22534666?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/MatthewR2D2/Pytorch/tree/master/GANS" target="_blank" rel="nofollow noopener">MatthewR2D2/Pytorch</a></h3>

<p>This repository is for my learning Pytorch. Contribute to MatthewR2D2/Pytorch development by creating an account on GitHub.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

username: Thabang_Lukhetho
cooked: <p>The image I’m generating are a sequence, that is why I’m using an LSTM, but I will have a look  the GAN repo thanks.</p>

username: JakeAndFinn
cooked: <p>GAN framework may be used with almost any kind of neural networks. Take a look at this papers for example:</p>
<ul>
<li>Continuous recurrent neural networks with adversarial training <a href="https://arxiv.org/abs/1611.09904" rel="nofollow noopener">https://arxiv.org/abs/1611.09904</a>
</li>
<li>Real-valued (Medical) Time Series Generation with Recurrent Conditional GANs <a href="https://arxiv.org/abs/1706.02633" rel="nofollow noopener">https://arxiv.org/abs/1706.02633</a>
</li>
<li>Language Generation with Recurrent Generative Adversarial Networks without Pre-training <a href="https://arxiv.org/abs/1706.01399" rel="nofollow noopener">https://arxiv.org/abs/1706.01399</a>
</li>
<li>Contextual RNN-GANs for Abstract Reasoning Diagram Generation <a href="https://arxiv.org/abs/1609.09444" rel="nofollow noopener">https://arxiv.org/abs/1609.09444</a>
</li>
</ul>

username: G.M
cooked: <p>First, it works for sure.<br>
Second, even though I’m unsure of this process, I think you can view the image generating lstm as the decoder lstm from Seq2Seq from NLP. <a href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#sphx-glr-intermediate-seq2seq-translation-tutorial-py" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#sphx-glr-intermediate-seq2seq-translation-tutorial-py</a> Your current approach might work as well; you can test the result different approaches.<br>
Third, you don’t have to initialize the hidden states to zeros manually, Pytorch lstm does that automatically.</p>

username: Thabang_Lukhetho
cooked: <p>Thank you very much for your help.</p>

username: Thabang_Lukhetho
cooked: <p>What do you mean by " it works for sure" you mean for my specified problem or it work in general?</p>

username: G.M
cooked: <p>Generating an image step by step using an lstm can be generalized to a sequence generating problem, and lstm is a common choice for generating sequences.</p>

username: Thabang_Lukhetho
cooked: <p>How would you configure your data if you had a inputs data set with the following dimension (batch_size=256, channels=2, height=64, width=64)  to match the LSTM input format (sequence_length, batch_size, input_size)  ?</p>

username: G.M
cooked: <p>You would first have to convert the image into a series of data; for example, flatten the image in some way(or flatten it using <code>Unfold</code>). Then reshape the tensor into the shape that match the input.</p>

username: Rahul_Gulia
cooked: <p>I am also trying to generate new images from a sequence of images using LSTM to learn and create future images. Please let me know if LSTM worked for you or if you used something else ?</p>