RagSequenceForGeneration without retriever
Author: raphaelsty
Pull request: no

Hi Hugging Face Team. 

First of all thank you for your work, I am a fan of Transformers. 😉

I'm opening this issue because I'm having trouble using the RagSequenceForGeneration, model that I'm particularly interested in. I do not want to use the default retriever but rather apply the generator on an input list of documents (retrieved using Elasticsearch). I have tried to dig into the code without success. I believe that the parameters are not correctly passed within the generator. I may have been mistaken about the use of the model API. I am looking for your help to generate text without using the retriever.  

## Environment info

- `transformers` version: 4.16.0.dev0
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.5
- PyTorch version (GPU?): 1.7.1 (False)
- Tensorflow version (GPU?): 2.4.1 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?:  False
- Using distributed or parallel set-up in script?: False

### Who can help
@patrickvonplaten, @narsil

Models:

- RAG generator

## Information

I'm using RagSequenceForGeneration and RagTokenForGeneration.

## To reproduce

Steps to reproduce the behavior:

```python
from transformers import RagTokenizer, RagTokenForGeneration, RagSequenceForGeneration
import torch

tokenizer = RagTokenizer.from_pretrained("facebook/rag-sequence-nq") 

model = RagSequenceForGeneration.from_pretrained(
    "facebook/rag-sequence-nq", retriever=None) 
 
q = "What is the capital of France ?"

documents = [
    "Paris is the capital of France",
    "Madrid is the capital of Spain",
    "Hugging Face rocks :)",
]

with tokenizer.as_target_tokenizer():

    query = tokenizer(q, return_tensors="pt")

    target = tokenizer(
        text = documents, 
        return_tensors = "pt", 
        padding = "longest"
    )

doc_scores = torch.tensor([[0.4], [0.3], [0.3]])

generated = model.generate(
    input_ids = query["input_ids"],
    attention_mask = query["attention_mask"],
    context_input_ids = target["input_ids"],
    context_attention_mask = target["attention_mask"],
    doc_scores = doc_scores,
    n_docs = 1,
)

tokenizer.batch_decode(
    generated,
    skip_special_tokens = False, 
    clean_up_tokenization_spaces = True, 
)
```

```python
AssertionError                            Traceback (most recent call last)
<ipython-input-3-23d39672d326> in <module>
     28 doc_scores = torch.tensor([[0.4], [0.3], [0.3]])
     29 
---> 30 generated = model.generate(
     31     input_ids = query["input_ids"],
     32     attention_mask = query["attention_mask"],

~/opt/miniconda3/lib/python3.8/site-packages/torch/autograd/grad_mode.py in decorate_context(*args, **kwargs)
     24         def decorate_context(*args, **kwargs):
     25             with self.__class__():
---> 26                 return func(*args, **kwargs)
     27         return cast(F, decorate_context)
     28 

~/opt/miniconda3/lib/python3.8/site-packages/transformers/models/rag/modeling_rag.py in generate(self, input_ids, attention_mask, context_input_ids, context_attention_mask, doc_scores, do_deduplication, num_return_sequences, num_beams, n_docs, **model_kwargs)
   1022             if input_ids is not None:
   1023                 new_input_ids = input_ids[index : index + 1].repeat(num_candidates, 1)
-> 1024                 outputs = self(new_input_ids, labels=output_sequences, exclude_bos_score=True)
   1025             else:  # input_ids is None, need context_input_ids/mask and doc_scores
   1026                 assert (

~/opt/miniconda3/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

~/opt/miniconda3/lib/python3.8/site-packages/transformers/models/rag/modeling_rag.py in forward(self, input_ids, attention_mask, encoder_outputs, decoder_input_ids, decoder_attention_mask, past_key_values, context_input_ids, context_attention_mask, doc_scores, use_cache, output_attentions, output_hidden_states, output_retrieved, exclude_bos_score, reduce_loss, labels, n_docs, **kwargs)
    846             use_cache = False
    847 
--> 848         outputs = self.rag(
    849             input_ids=input_ids,
    850             attention_mask=attention_mask,

~/opt/miniconda3/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

~/opt/miniconda3/lib/python3.8/site-packages/transformers/models/rag/modeling_rag.py in forward(self, input_ids, attention_mask, encoder_outputs, decoder_input_ids, decoder_attention_mask, past_key_values, doc_scores, context_input_ids, context_attention_mask, use_cache, output_attentions, output_hidden_states, output_retrieved, n_docs)
    656                     ).squeeze(1)
    657             else:
--> 658                 assert (
    659                     context_input_ids is not None
    660                 ), "Make sure that `context_input_ids` are passed, if no `retriever` is set. Alternatively, you can set a retriever using the `set_retriever(...)` function."

AssertionError: Make sure that `context_input_ids` are passed, if no `retriever` is set. Alternatively, you can set a retriever using the `set_retriever(...)` function.
```

## Expected behavior

```python
# q = "What is the capital of France ?"
["Paris"]
```


Comments:
Hi @raphaelsty I am definitely not familiar with Rag.

Looking briefly at the stack, there is indeed no `context_input_ids` passed to  (line 1024), I tried to manually add them just to see, and other errors pop up (linked to tensor dimensions).
Since I am not super familiar I can't say if we shouldn't end in that branch, or if the code has indeed a bug.

Do you mind sharing where you have adapted your code from ?

Pinging @lhoestq which is also responsible for Rag apparently.


I can take a look at it :-)

Hi @Narsil, @patrickvonplaten, thank you for your quick response,

I have just managed to use the RagTokenForGeneration without the retriever. 

I had not correctly formed the `context_input_ids` and `context_attention_mask` fields in my previous post. The code below works correctly for me. 

I relied on the [documentation](https://huggingface.co/docs/transformers/model_doc/rag#transformers.TFRagTokenForGeneration.generate.doc_scores) to write the code but I have found an example of RAG without the retriever in [Haystack](https://github.com/deepset-ai/haystack/blob/c6f23dce8897ab00fcb15e272282d459dcfa564a/haystack/nodes/answer_generator/transformers.py#L151).

```python
from transformers import RagTokenizer, RagTokenForGeneration, RagSequenceForGeneration, RagRetriever
import torch

tokenizer = RagTokenizer.from_pretrained("facebook/rag-token-nq") 

model = RagTokenForGeneration.from_pretrained(
    "facebook/rag-token-nq", retriever=None) 

num_beams = 10
k = 2

q = "What is the capital of France ?"

documents = [
    "The capital of germany is Berlin",
    "The capital of France is Paris",
]

q_documents = [
    f"{document} {model.config.doc_sep} {q}" for document in documents
]

context = tokenizer.generator.batch_encode_plus(
    q_documents,
    max_length= model.config.max_combined_length,
    return_tensors="pt",
    padding="max_length",
    truncation=True,
)

doc_scores = torch.tensor([[0.9, 0.1]], dtype=torch.float)

# Get generated ids from generator
generator_ids = model.generate(
    context_input_ids = context["input_ids"], 
    context_attention_mask = context["attention_mask"],
    doc_scores = doc_scores,
    num_return_sequences=k,
    num_beams=min(num_beams, k),
    max_length=40,
    min_length=1,
    n_docs=len(documents)
)

tokenizer.batch_decode(generator_ids, skip_special_tokens=True)
[' paris', " `` paris ''"]
```

I think we can close this issue :)



Thanks a lot for posting a sample for future readers