Tqdm not working with question-answering pipeline
Author: Alex-apostolo
Pull request: no
Labels: name: bug

### System Info

Running inside a notebook on Google Colab
- `transformers` version: 4.21.1
- Platform: Linux-5.4.188+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- Huggingface_hub version: 0.8.1
- PyTorch version (GPU?): 1.12.0+cu113 (True)
- Tensorflow version (GPU?): 2.8.2 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@Narsil, @sgugger

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Run the below code:

from tqdm.auto import tqdm
from transformers import pipeline

dataset = load_dataset('cuad', split='test')
dataset = dataset.remove_columns(['id', 'title', 'answers'])
nlp = pipeline("question-answering", device=0, batch_size=64)

for answer in tqdm(nlp(dataset)):
    print(answer)

### Expected behavior

Expected to see a progress bar with the time remaining and the it/s but nothing is displayed. 

Comments:
Hi there,

Try this:

```python
from tqdm.auto import tqdm
from transformers import pipeline
from datasets import load_dataset

dataset = load_dataset('cuad', split='test')
dataset = dataset.remove_columns(['id', 'title', 'answers'])

batch_size = 16
nlp = pipeline("question-answering", device=0)

results = []
for i in tqdm(range(0, len(dataset), batch_size)):
    results.extend(
        nlp(
            context=dataset[i:i+batch_size]["context"],
            question=dataset[i:i+batch_size]["question"]
        )
    )
```

@nbroad1881 Thanks a bunch that worked! 

@Alex-apostolo ,

The answer from @nbroad1881 will work, however it will not batch anything because the pipeline was not set with batching.

```python
from tqdm.auto import tqdm
from transformers import pipeline
from datasets import load_dataset

dataset = load_dataset('cuad', split='test')
dataset = dataset.remove_columns(['id', 'title', 'answers'])

batch_size = 16
nlp = pipeline("question-answering", device=0, batch_size=batch_size)  # <--- small change

results = []
for i in tqdm(range(0, len(dataset), batch_size)):
    results.extend(
        nlp(
            context=dataset[i:i+batch_size]["context"],
            question=dataset[i:i+batch_size]["question"]
        )
    )
```

This should work more as you intend.
Keep in mind that batching will occur on chunks of text, not on the entire question/context. That's a feature since you have more control on the memory + sequence_length of what the model sees. So while you are sending 16 question+context pairs at a time you might get any amount of forward calls depending on the chunking of those pairs (only 1 forward call if the pair is small enough). `max_seq_len`,  `doc_stride` and `max_question_len` might have to be adjusted for your dataset+model pair. (There are defaults used for squad, but it might impact actual score for your use case).


Actually the problem is not really the pipeline in general it should work with `tqdm`. The problem is the legacy support for many args that's actually looking a tthe whole dataset to create `SquadExample` out of it. 
I am going to look at solutions for this, since consuming the entire dataset before feeding it to the pipeline (+ consuming memory) is not really intended.

```python
from tqdm.auto import tqdm
from transformers import pipeline
from datasets import load_dataset

dataset = load_dataset("cuad", split="test")
dataset = dataset.remove_columns(["id", "title", "answers"])

pipe = pipeline("question-answering", device=0, batch_size=1, framework="pt")


def data(dataset):
    for item in dataset:
        yield {"question": item["question"], "context": item["context"]}


results = []
for out in tqdm(pipe(data(dataset)), total=len(dataset)):
    pass  # print(out)
```
Here is an example that should be working + desirable (it actually works, but is *not* an iterator like intended.