torch.nn.Embedding() for text2image generation

bekhzod: <p>I am trying to generate an image of a license plate number using text input. For example, if an input is 04누0839, the model should return the following image.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/8/1/8182837c3320994286206430e8c540191ea91f79.jpeg" alt="04누0839" data-base62-sha1="itHdvhFpPHwalhguvOAFtoy7Gpr" width="328" height="76"></p>
<p>I am using <a href="https://github.com/lucidrains/imagen-pytorch" rel="noopener nofollow ugc">imagen-pytorch</a> but it does not output satisfactory images. When trained using “unconditional Imagen” (without text embeddings), the output images are fair enough but input text is ignored.</p>
<p>When trained using text_embeddings (torch.nn.Embedding()):</p>
<pre><code class="lang-auto">word_set = [text for text in self.texts]
vocab = {tkn: i for i, tkn in enumerate(word_set)}
torch.manual_seed(2023)
embedding_layer = nn.Embedding(len(vocab), 1)
self.text_embs = []
for num in tqdm(vocab.keys()):    
    lookup_tensor = torch.tensor([vocab[num]], dtype=torch.long)
    embedding = embedding_layer(lookup_tensor)
    self.text_embs.append(embedding)
    # print(text_embs.shape)
self.text_embeds = torch.stack(self.text_embs)
</code></pre>
<p>the outputs are very unsatisfactory.</p>
<p>I think I use Embedding layer improperly. Here are my questions:</p>
<ol>
<li>How to use it correctly? Should I increase embedding_dim? (I am using 1 now).</li>
<li>How about manual_seed? (Training with and without manual seed did not improve the performance, though).</li>
<li>Is the embedding layer trained automatically when I train imagen or I should train it first on my data and then make embeddings as input to the imagen?</li>
<li>Should I split the text (ex. 04누0839 as ‘0’, ‘4’, ‘누’…) and then create embeddings or it is fine to use the whole text to create embeddings?</li>
</ol>

J_Johnson: <p>An embedding layer provides a learnable vectorized set of weights to turn tokens into vectors. The nice thing about vectors is they can store information about how various words relate to one another. Like coffee and water might score closer in more vector points than coffee and hamburger, after training, of course.</p>
<p>Here is a good tutorial on <code>nn.Embedding</code>:</p>
<p><a href="https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html</a></p>
<p>One thing I noticed in your code is that you are re-indexing any repeated words in self.text. Meaning that the same word may be indexed twice or more.</p>
<p>Additionally, the context which you’re applying <code>nn.Embedding</code> may be ill-suited. That’s because 0 and 4, for example, are very unilateral in their relationship with one another. I.e. you wouldn’t be able to describe either in terms of liquid, solid, gas, or verb, noun, etc. They do not have properties outside of their representation as a numerical identifier on a car plate.</p>

bekhzod: <p>Thank you, <a class="mention" href="/u/j_johnson">@J_Johnson</a>! Can you please tell me how I should represent numerical identifiers (digits)?</p>

J_Johnson: <p>Image-gen isn’t specifically trained on diffusing into license plate, so your results will likely be hit and miss. If you want something like that, perhaps try starting with training a stable diffusion model. Something like this:</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/CompVis/stable-diffusion">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/CompVis/stable-diffusion" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/345;"><img src="https://opengraph.githubassets.com/7985c6b7d0ca882ed255dd9c8a92e2198809c24732fd5e79382fc70be2488ee2/CompVis/stable-diffusion" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/CompVis/stable-diffusion" target="_blank" rel="noopener nofollow ugc">GitHub - CompVis/stable-diffusion: A latent text-to-image diffusion model</a></h3>

  <p>A latent text-to-image diffusion model. Contribute to CompVis/stable-diffusion development by creating an account on GitHub.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Anyway, to your question, I’d retrain a new text encoder. With less than 100 chars, you could probably just pass in a <code>one_hot</code> embedding, assigning a given row vector to each letter/number, with a length of the max number of letter/numbers available.</p>
<p>The encoder should be very small, no more than 1M parameters for quicker training time. Likewise, the UNet should be small. You could likely remove a downsampling/upsampling layer or two since you probably don’t need more than HxW=64x32 pixel images as outputs. I would keep it under 20M parameters. That way you can train it on an Nvidia 3xxx or 4xxx in a day.  Otherwise, you’ll need a big budget and lots of time to train a full UNet on denoising diffusion.</p>