Repository: flairNLP/flair

Issue:
Generate text with language model
Is there any script that can generate text from trained language model?
Also calculating perplexity on testing data would be helpful.


Discussion:
Hi @djstrong you can point the following script to a language model that you've trained with Flair: 

```python
import torch
from flair.models import LanguageModel

dataset = '/path/to/your/lm/best-lm.pt'

# load the language model
model = LanguageModel.load_language_model(dataset)

idx2item = model.dictionary.idx2item

# initial hidden state
hidden = model.init_hidden(1)
input = torch.rand(1, 1).mul(len(idx2item)).long()

# generate text character by character
characters = []
number_of_characters_to_generate = 2000
for i in range(number_of_characters_to_generate):
    prediction, rnn_output, hidden = model.forward(input, hidden)
    word_weights = prediction.squeeze().data.div(1.0).exp().cpu()
    word_idx = torch.multinomial(word_weights, 1)[0]
    input.data.fill_(word_idx)
    word = idx2item[word_idx].decode('UTF-8')
    characters.append(word)

    if i % 100 == 0:
        print('| Generated {}/{} chars'.format(i, number_of_characters_to_generate))

# print generated text
print(''.join(characters))
```

This generates 2000 characters. I think in a future release, we will add such code directly to the `LanguageModel` class to make this easier! 
Thank you, it works. However, after more than 100 characters there is `CUDA error: out of memory`.
Edit: A solution to this problem is `with torch.no_grad():` before the loop.
Ah, thanks for this! In fact, [looks like](https://discuss.pytorch.org/t/model-eval-vs-with-torch-no-grad/19615/5) we might want to add this in other parts of the framework as well, since autograd is not needed at evaluation or prediction time.   
@alanakbik I tried an own language model, but the generation method above gives the following error message:

```bash
Traceback (most recent call last):
  File "generate.py", line 19, in <module>
    prediction, rnn_output, hidden = model.forward(input, hidden)
  File "/mnt/flair/flair/models/language_model.py", line 66, in forward
    encoded = self.encoder(input)
  File "/.pyenv/versions/3.6.6/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/.pyenv/versions/3.6.6/lib/python3.6/site-packages/torch/nn/modules/sparse.py", line 110, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/.pyenv/versions/3.6.6/lib/python3.6/site-packages/torch/nn/functional.py", line 1110, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Expected object of type torch.cuda.LongTensor but found type torch.LongTensor for argument #3 'index'
```

Using the latest version from `master`, can you help?
Hello @stefan-it it looks like the model is on cuda while the tensors that are being passed are normal non-cuda tensors. There's two ways to address this: 

- Put the language model on cpu by calling .cpu() after you've initialized the model, i.e. 
`model = model.cpu()`
- Put the input tensor to cuda. You can do this by calling .cuda() on the input tensor when you create it, i.e. 
`input = torch.rand(1, 1).mul(len(idx2item)).long().cuda()`

Does this work for you? When we add this method officially, we'll work out all the cuda stuff :)
Thanks Alan, this is working for me:

```python
import torch
from flair.models import LanguageModel

dataset = 'best-lm.pt'

# load the language model
model = LanguageModel.load_language_model(dataset)

model = model.cuda()

idx2item = model.dictionary.idx2item

# initial hidden state
hidden = model.init_hidden(1)
input = torch.rand(1, 1).mul(len(idx2item)).long().cuda()

# generate text character by character
characters = []
number_of_characters_to_generate = 500
for i in range(number_of_characters_to_generate):
    prediction, rnn_output, hidden = model.forward(input, hidden)
    word_weights = prediction.squeeze().data.div(1.0).exp().cpu()
    word_idx = torch.multinomial(word_weights, 1)[0]
    input.data.fill_(word_idx)
    word = idx2item[word_idx].decode('UTF-8')
    characters.append(word)

    if i % 100 == 0:
        print('| Generated {}/{} chars'.format(i, number_of_characters_to_generate))

# print generated text
print(''.join(characters))
```

Resolving patch:
diff --git a/flair/models/language_model.py b/flair/models/language_model.py
--- a/flair/models/language_model.py
+++ b/flair/models/language_model.py
@@ -2,7 +2,7 @@
 import torch
 import math
 from torch.autograd import Variable
-from typing import Dict, List
+from typing import List
 from flair.data import Dictionary
 
 
@@ -148,3 +148,24 @@ def save(self, file):
             'dropout': self.dropout
         }
         torch.save(model_state, file, pickle_protocol=4)
+
+    def generate_text(self, number_of_characters=1000) -> str:
+        characters = []
+
+        idx2item = self.dictionary.idx2item
+
+        # initial hidden state
+        hidden = self.init_hidden(1)
+        input = torch.rand(1, 1).mul(len(idx2item)).long()
+        if torch.cuda.is_available():
+            input = input.cuda()
+
+        for i in range(number_of_characters):
+            prediction, rnn_output, hidden = self.forward(input, hidden)
+            word_weights = prediction.squeeze().data.div(1.0).exp().cpu()
+            word_idx = torch.multinomial(word_weights, 1)[0]
+            input.data.fill_(word_idx)
+            word = idx2item[word_idx].decode('UTF-8')
+            characters.append(word)
+
+        return ''.join(characters)


Test patch: