Accelerate preprocessing crashing due to non-tensor input
Author: ElleLeonne
Pull request: no

### System Info

I believe a recent update has caused Accelerate to try and concatenate all tensor data in the input dictionary.

This is a problem, because my inputs contain non-tensor data, due to the fact that such data is intermittent and not always provided in the batch.

Rather than skipping over this information, it instead tries to torch.cat the data, which results in crashes.

```
Traceback (most recent call last):
  File "/home/lily/Desktop/Project/finetune_dynamic.py", line 304, in <module>
    fire.Fire(train)
  File "/home/lily/anaconda3/envs/emme/lib/python3.10/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/home/lily/anaconda3/envs/emme/lib/python3.10/site-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/home/lily/anaconda3/envs/emme/lib/python3.10/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/home/lily/Desktop/Emme (copy)/finetune_dynamic.py", line 295, in train
    trainer.train(resume_from_checkpoint=resume_from_checkpoint)
  File "/home/lily/anaconda3/envs/emme/lib/python3.10/site-packages/transformers/trainer.py", line 1539, in train
    return inner_training_loop(
  File "/home/lily/anaconda3/envs/emme/lib/python3.10/site-packages/transformers/trainer.py", line 1779, in _inner_training_loop
    for step, inputs in enumerate(epoch_iterator):
  File "/home/lily/anaconda3/envs/emme/lib/python3.10/site-packages/accelerate/data_loader.py", line 553, in __iter__
    next_batch, next_batch_info = self._fetch_batches(main_iterator)
  File "/home/lily/anaconda3/envs/emme/lib/python3.10/site-packages/accelerate/data_loader.py", line 521, in _fetch_batches
    batch = concatenate(batches, dim=0)
  File "/home/lily/anaconda3/envs/emme/lib/python3.10/site-packages/accelerate/utils/operations.py", line 413, in concatenate
    return type(data[0])({k: concatenate([d[k] for d in data], dim=dim) for k in data[0].keys()})
  File "/home/lily/anaconda3/envs/emme/lib/python3.10/site-packages/accelerate/utils/operations.py", line 413, in <dictcomp>
    return type(data[0])({k: concatenate([d[k] for d in data], dim=dim) for k in data[0].keys()})
  File "/home/lily/anaconda3/envs/emme/lib/python3.10/site-packages/accelerate/utils/operations.py", line 411, in concatenate
    return honor_type(data[0], (concatenate([d[i] for d in data], dim=dim) for i in range(len(data[0]))))
  File "/home/lily/anaconda3/envs/emme/lib/python3.10/site-packages/accelerate/utils/operations.py", line 84, in honor_type
    return type(obj)(generator)
  File "/home/lily/anaconda3/envs/emme/lib/python3.10/site-packages/accelerate/utils/operations.py", line 411, in <genexpr>
    return honor_type(data[0], (concatenate([d[i] for d in data], dim=dim) for i in range(len(data[0]))))
  File "/home/lily/anaconda3/envs/emme/lib/python3.10/site-packages/accelerate/utils/operations.py", line 411, in concatenate
    return honor_type(data[0], (concatenate([d[i] for d in data], dim=dim) for i in range(len(data[0]))))
  File "/home/lily/anaconda3/envs/emme/lib/python3.10/site-packages/accelerate/utils/operations.py", line 84, in honor_type
    return type(obj)(generator)
  File "/home/lily/anaconda3/envs/emme/lib/python3.10/site-packages/accelerate/utils/operations.py", line 411, in <genexpr>
    return honor_type(data[0], (concatenate([d[i] for d in data], dim=dim) for i in range(len(data[0]))))
  File "/home/lily/anaconda3/envs/emme/lib/python3.10/site-packages/accelerate/utils/operations.py", line 415, in concatenate
    raise TypeError(f"Can only concatenate tensors but got {type(data[0])}")
TypeError: Can only concatenate tensors but got <class 'str'>
```
I'd like to know if there's a way to turn off this feature in accelerate. I can (and have been, up until now) handle batching my own data. It's only now that it's become a problem.

Issue raised on accelerate as well:
https://github.com/huggingface/accelerate/issues/1611

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Reproduce:

1: Create model that accepts non-tensor inputs as a nested list
2: Feed input to model via huggingface trainer
3: Observe crash

### Expected behavior

Accelerate should ignore the data that it can't process, and pass it to the model as normal.

Comments:
Hi @ElleLeonne, 

So that we can best help, could you share a minimal code snippet so that we can reproduce the error and information about the running environment: run `transformers-cli env` in the terminal and copy-paste the output?

Out of town today, but I'll have something shortly.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

I'm also getting this error when trying to use data that isn't a tensor. @ElleLeonne did you ever find a solution to this?

@irowberryFS If you could provide a reproducer for this issue, the full error traceback and information about the running environment then we'll be able to look into it. 

@amyeroberts I couldn't reproduce it using a small training script. For the small script I am using PyTorch Geometric datasets, and their DataLoaders and Accelerator worked fine. Unfortunately all the other threads about this issue don't contain any snippets of training code. I believe it has to do with the `IterableDataset` as described in this issue #26548 . I have temporarily bypassed the issue by not wrapping the DataLoader in the `accelerator.prepare()` call. Here is my dataset processing code. 

```
dataset = load_dataset("parquet", data_dir="../../mobTrain/", streaming=True)['train']
dataset = dataset.shuffle(seed=42)

def convert_mobs(row):
    row['mob1'] = process_mob(row['mob1'])
    row['mob2'] = process_mob(row['mob2'])
    return row

dataset = dataset.map(convert_mobs, remove_columns=['...', '...'])
dl = DataLoader(dataset, batch_size)
```
Here is the traceback
```
Traceback (most recent call last):
  File "/home/username/projects/GraphBlocking/train_model.py", line 121, in <module>
    main()
  File "/home/username/projects/GraphBlocking/train_model.py", line 118, in main
    train()
  File "/home/username/projects/GraphBlocking/train_model.py", line 93, in train
    for batch in tepoch:
  File "/home/username/projects/venv/lib/python3.11/site-packages/tqdm/std.py", line 1178, in __iter__
    for obj in iterable:
  File "/home/username/projects/venv/lib/python3.11/site-packages/accelerate/data_loader.py", line 639, in __iter__
    next_batch, next_batch_info = self._fetch_batches(main_iterator)
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/username/projects/venv/lib/python3.11/site-packages/accelerate/data_loader.py", line 602, in _fetch_batches
    batch = concatenate(batches, dim=0)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/username/projects/venv/lib/python3.11/site-packages/accelerate/utils/operations.py", line 530, in concatenate
    return type(data[0])({k: concatenate([d[k] for d in data], dim=dim) for k in data[0].keys()})
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/username/projects/venv/lib/python3.11/site-packages/accelerate/utils/operations.py", line 530, in <dictcomp>
    return type(data[0])({k: concatenate([d[k] for d in data], dim=dim) for k in data[0].keys()})
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/username/projects/venv/lib/python3.11/site-packages/accelerate/utils/operations.py", line 532, in concatenate
    raise TypeError(f"Can only concatenate tensors but got {type(data[0])}")
TypeError: Can only concatenate tensors but got <class 'torch_geometric.data.batch.HeteroDataBatch'>
```
The `DataLoader` is a PyG wrapper of the PyTorch `DataLoader`. I access the data in the typical way of `for batch in dataloader:` Again, I believe the issue is using Accelerate with an `IterableDataset`

@irowberryFS Have you tried any the suggestions in [this comment](https://github.com/huggingface/transformers/issues/26548#issuecomment-1885798533) or the rest of the thread you linked to? 

@amyeroberts yes. I set `dispatch_batches=False` and it fixed the issue. 

@irowberryFS Thanks for reporting back! 

I think the cause is that [accelerate.utils.concatenate](https://github.com/huggingface/accelerate/blob/v1.6.0/src/accelerate/utils/operations.py#L596) forces all elements to be returned as tensor. Maybe this restriction should be loosened?