opened
rajarsheem
username_0
Hi,

I have raised this issue before but unfortunately that didn't help me.
I have a model which uses elmo and after training my model (fine-tuning elmo alongside), I load my trained model.

```python
    model.eval()
    state = T.load("model.pth")
    model.load_state_dict(state['state_dict'])
```

Then I try to check the embedding of the fine-tuned elmo from inside the trained model. I get the embedding on a sample input repeatedly and print the embedding sum after every 10 steps. The sum is varying a lot which is consequently changing my model predictions. I have tried to follow the notes on statefulness but that didn't help either.

```python
    for step in range(100):
        # embedding = model.elmo(query_[:3])['elmo_representations'][0]
        # gs = model(query_, good_)
        user_query = 'uses of iphone 4s'.split(' ')
        embedding = model.elmo(batch_to_ids([user_query]).to(T.device(args.device)))['elmo_representations'][0]
        if step % 10 == 0:
            print(embedding.sum())
```

`tensor(-63.9617, device='cuda:3', grad_fn=)                                                                                                                                          tensor(-32.3701, device='cuda:3', grad_fn=)                                                                                                                                          tensor(-59.4102, device='cuda:3', grad_fn=)                                                                                                                                          tensor(-56.8035, device='cuda:3', grad_fn=)                                                                                                                                          tensor(-31.6015, device='cuda:3', grad_fn=)                                                                                                                                          tensor(-24.1763, device='cuda:3', grad_fn=)                                                                                                                                          tensor(-25.7312, device='cuda:3', grad_fn=)                                                                                                                                          tensor(-25.7935, device='cuda:3', grad_fn=)                                                                                                                                          tensor(-25.0103, device='cuda:3', grad_fn=)                                                                                                                                          tensor(-24.9410, device='cuda:3', grad_fn=)`




As you can see from the o/p, there is a inconsistency with the embedding returned. Can you please help?
embedding is different on same input after loading from fine-tuned model
issue
created
nelson-liu
username_1
have you seen this? https://github.com/allenai/allennlp/blob/master/tutorials/how_to/elmo.md#notes-on-statefulness-and-non-determinism

seems expected to me, since your context is being filled up with `"uses of iphone 4s"` as you iterate
comment
created
rajarsheem
username_0
Yes, I have followed that but warming up (if I am doing that right) didn't help. Isn't there a way to make the embedding consistent?
comment
created
matt-gardner
username_2
Theoretically, the RNN is stateful, and in general will have output that depends on the entire history of what it has seen before (i.e., not consistent). There's no way around that, theoretically. Empirically, because of the way ELMo is trained, we observe consistency across inputs after a warmup period, though this is not guaranteed theoretically. It seems likely that your fine-tuning of it upset that balance somehow, for some reason that no one really understands at this point.

That said, you could manually reset the RNN state in between all of your predictions to guarantee consistent embeddings. I don't have a principled value to recommend resetting the state to, though.
comment
created
rajarsheem
username_0
Okay, so do you mean no matter what the inputs are, the embedding of my 5th training example (a query) will be dependent on the previous 4 examples ? Is it preserving the state across training examples or batches?
comment
created
rajarsheem
username_0
So, it is preserving the state across examples. Same batch with changed order causes different embedding in the next run.
comment
created
rajarsheem
username_0
Hey I just discovered that `_EncoderBase` has attribute `stateful` which when set to False can prevent preserving the state and zeroes out the initial state between forwards.

It seems to solve my issue. Is this recommended?
comment
created
matt-gardner
username_2
As I said, no one really understands what's going on inside these RNNs, so there is no "recommended" setup if you've fine tuned an ELMo model. This is an open research area. Just do whatever you think works best for what you want to do with it.
comment
created
rajarsheem
username_0
Agreed. But turning off `stateful` resets the lstm states after every forward pass. Is that understanding correct? 

If this doesn't have any other undesirable consequences somewhere else, then it solves my problem and also closes this issue!
comment
closed
matt-gardner
username_2
issue
created
matt-gardner
username_2
Yes, that's what the `stateful` parameter does.  Turning it off will mean that your model gets different input at test time than you gave it at training time, which could adversely affect your performance.  But, that's something that you'll have to evaluate yourself - I have no intuition for how much it will matter.

I'm glad that this solves your problem, and I'm closing the issue.  Feel free to re-open, or open a new one, if you have more issues.
comment
created
matt-peters
username_3
I'd strongly recommend checking performance before / after switching the `stateful` flag.  Without fine tuning, switching `stateful=False` will likely have an adverse impact on performance of ELMo.  After fine tuning who knows as MattG said.
comment
created
rajarsheem
username_0
@username_3 : 
The reason I need to keep the model as stateless because my task represents like a standard sequence classification task where I need to encode batch of sentences and finally predict a label for each of them.

As you can understand, embedding of 4th sentence in the batch should be independent of the previous examples (1 to 3) in the batch. Setting `stateful=True` preserves the LSTM states across all the examples in the batch (from what I understand).

Regarding performance post setting `stateful=False`, I have seen that the convergence is in progress but does not reach a satisfactory minima. Now that might be also because of the inefficiency of my model and not the flag change in elmo.

My question is how do I use the elmo for any standard  non-LM task where all the examples in the batch are independent of each other? Does not any such task requires model to be stateless in terms of moving from one example to another example?
comment
created
matt-peters
username_3
Since the language model was trained on shuffled sentences, the forward language model learned to reset its own internal states after seeing the token sequence `</S> <S>` (end of sentence, beginning of sentence).  The backward model does something similar after seeing `<S> </S>`.  These tokens are added automatically by the allennlp code, so as long as each instance in your batch is one complete sentence / paragraph / chunk of context they will be essentially treated as independent.

In practice, we haven't noticed any adverse impact from the statefulness after the first few batches once the model has been given the opportunity to reset states from their initial 0 values.

Something about your fine tuning process is adversely impacting the resetting of the states.  If I re-run your example with the pretrained ELMo model, then the embeddings quickly stability so that they are constant after the first few batches.

```
from allennlp.commands.elmo import ElmoEmbedder

elmo = ElmoEmbedder(options_file, weight_file)

tokens = 'uses of iphone 4s'.split(' ') 
for step in range(100):
    embeddings = elmo.embed_sentence(tokens)
    # embeddings[-1] is the top layer
    if step % 10 == 0:
        print(step, embeddings[-1].sum())
```
prints:

```
01/30/2019 13:25:51 - INFO - allennlp.commands.elmo -   Initializing ELMo.
0 -8.709412
10 -12.381893
20 -12.381773
30 -12.381746
40 -12.381765
50 -12.381748
60 -12.381767
70 -12.381752
80 -12.381767
90 -12.38179
```
comment