How to convert  torch.utils.data.Dataset to huggingface dataset?
State: closed

I look through the huggingface dataset docs, and it seems that there is no offical support function to  convert  `torch.utils.data.Dataset`  to huggingface dataset. However, there is a way to convert huggingface dataset to  `torch.utils.data.Dataset`, like below:
```python
from datasets import Dataset
data = [[1, 2],[3, 4]]
ds = Dataset.from_dict({"data": data})
ds = ds.with_format("torch")
ds[0]
ds[:2]
```
So is there something I miss, or there IS no function to convert  `torch.utils.data.Dataset` to huggingface dataset. If so, is there any way to do this convert?
Thanks.

Hi! I think you can use the newly-added `from_generator` method for that:
```python
from datasets import Dataset

def gen():
    for idx in len(torch_dataset):
        yield torch_dataset[idx]  # this has to be a dictionary
    ## or if it's an IterableDataset
    # for ex in torch_dataset:
    #     yield ex

dset = Dataset.from_generator(gen)
```

Maybe `Dataset.from_list` can work as well no ?
```python
from datasets import Dataset

dset = Dataset.from_list(torch_dataset)
```

> ```python
> from datasets import Dataset
> 
> def gen():
>     for idx in len(torch_dataset):
>         yield torch_dataset[idx]  # this has to be a dictionary
>     ## or if it's an IterableDataset
>     # for ex in torch_dataset:
>     #     yield ex
> 
> dset = Dataset.from_generator(gen)
> ```

I try to use `Dataset.from_generator()` method, and it returns an error:
```bash
AttributeError: type object 'Dataset' has no attribute 'from_generator'
```
And I think it maybe the version of my datasets package is out-of-date, so I update it
```bash
pip install --upgrade datasets
```
But after that, the code still return the above Error. 

> ```python
> dset = Dataset.from_list(torch_dataset)
> ```

It seems that Dataset also has no `from_list` method 😂
```bash
AttributeError: type object 'Dataset' has no attribute 'from_list'
```

> I look through the huggingface dataset docs, and it seems that there is no offical support function to convert `torch.utils.data.Dataset` to huggingface dataset. However, there is a way to convert huggingface dataset to `torch.utils.data.Dataset`, like below:
> 
> ```python
> from datasets import Dataset
> data = [[1, 2],[3, 4]]
> ds = Dataset.from_dict({"data": data})
> ds = ds.with_format("torch")
> ds[0]
> ds[:2]
> ```
> 
> So is there something I miss, or there IS no function to convert `torch.utils.data.Dataset` to huggingface dataset. If so, is there any way to do this convert? Thanks.

My dummy code is like:
```python
import os
import json
from torch.utils import data
import datasets

def gen(torch_dataset):
    for idx in len(torch_dataset):
        yield torch_dataset[idx]  # this has to be a dictionary

class MyDataset(data.Dataset):
    def __init__(self, path):
        self.dict = []
        for line in open(path, 'r', encoding='utf-8'):
            j_dict = json.loads(line)
            self.dict.append(j_dict['context'])
            
    def __getitem__(self, idx):
        return self.dict[idx]

    def __len__(self):
        return len(self.dict)

root_path = os.path.dirname(os.path.abspath(__file__))
path = os.path.join(root_path, 'dataset', 'train.json')
torch_dataset = MyDataset(path)

dit = []
for line in open(path, 'r', encoding='utf-8'):
    j_dict = json.loads(line)
    dit.append(j_dict['context'])
dset1 = datasets.Dataset.from_list(dit)
print(dset1)
dset2 = datasets.Dataset.from_generator(gen)
print(dset2)
```

We're releasing `from_generator` and `from_list` today :)
In the meantime you can play with them by installing `datasets` from source

> We're releasing `from_generator` and `from_list` today :) In the meantime you can play with them by installing `datasets` from source

Thanks a lot for your work!

> > I look through the huggingface dataset docs, and it seems that there is no offical support function to convert `torch.utils.data.Dataset` to huggingface dataset. However, there is a way to convert huggingface dataset to `torch.utils.data.Dataset`, like below:
> > ```python
> > from datasets import Dataset
> > data = [[1, 2],[3, 4]]
> > ds = Dataset.from_dict({"data": data})
> > ds = ds.with_format("torch")
> > ds[0]
> > ds[:2]
> > ```
> > 
> > 
> >     
> >       
> >     
> > 
> >       
> >     
> > 
> >     
> >   
> > So is there something I miss, or there IS no function to convert `torch.utils.data.Dataset` to huggingface dataset. If so, is there any way to do this convert? Thanks.
> 
> My dummy code is like:
> 
> ```python
> import os
> import json
> from torch.utils import data
> import datasets
> 
> def gen(torch_dataset):
>     for idx in len(torch_dataset):
>         yield torch_dataset[idx]  # this has to be a dictionary
> 
> class MyDataset(data.Dataset):
>     def __init__(self, path):
>         self.dict = []
>         for line in open(path, 'r', encoding='utf-8'):
>             j_dict = json.loads(line)
>             self.dict.append(j_dict['context'])
>             
>     def __getitem__(self, idx):
>         return self.dict[idx]
> 
>     def __len__(self):
>         return len(self.dict)
> 
> root_path = os.path.dirname(os.path.abspath(__file__))
> path = os.path.join(root_path, 'dataset', 'train.json')
> torch_dataset = MyDataset(path)
> 
> dit = []
> for line in open(path, 'r', encoding='utf-8'):
>     j_dict = json.loads(line)
>     dit.append(j_dict['context'])
> dset1 = datasets.Dataset.from_list(dit)
> print(dset1)
> dset2 = datasets.Dataset.from_generator(gen)
> print(dset2)
> ```
Hi, when I am using this code to build my own dataset, ` datasets.Dataset.from_generator(gen)` report `TypeError: cannot pickle generator object` whre MyDataset returns a dict like {'image': bytes, 'text': string}. How can I resolve this? Thanks a lot!

Hi ! Right now generator functions are expected to be picklable, so that `datasets` can hash it and use the hash to cache the resulting Dataset on disk. Maybe this can be improved.

In the meantime, can you check that you're not using unpickable objects. In your case it looks like you're using a generator object that is unpickable. It might come from an opened file, e.g. this doesn't work:
```python
with open(...) as f:

    def gen():
        for x in f:
            yield json.loads(x)

    ds = Dataset.from_generator(gen)
```
but this does work:
```python
def gen():
    with open(...) as f:
        for x in f:
            yield json.loads(x)

ds = Dataset.from_generator(gen)
```

> Hi ! Right now generator functions are expected to be picklable, so that `datasets` can hash it and use the hash to cache the resulting Dataset on disk. Maybe this can be improved.
> 
> In the meantime, can you check that you're not using unpickable objects. In your case it looks like you're using a generator object that is unpickable. It might come from an opened file, e.g. this doesn't work:
> 
> ```python
> with open(...) as f:
> 
>     def gen():
>         for x in f:
>             yield json.loads(x)
> 
>     ds = Dataset.from_generator(gen)
> ```
> 
> but this does work:
> 
> ```python
> def gen():
>     with open(...) as f:
>         for x in f:
>             yield json.loads(x)
> 
> ds = Dataset.from_generator(gen)
> ```

Thanks a lot! That's the reason why I have encountered this issue. Sorry for bothering you again with another problem, since my dataset is large and I use IterableDataset.from_generator which has no attribute with_transform, how can I equip it with some customed preprocessings like Dataset.from_generator? Should I move the preprocessing to the my torch Dataset?

Iterable datasets are lazy: exactly like `with_transform` they apply processing on the fly when accessing the examples.

Therefore you can use `my_iterable_dataset.map()` instead :)

@lhoestq thanks a lot and I have successfully made it work~

@lhoestq  I am having a similar issue. Can you help me understand which kinds of generators are picklable? I previously thought that no generators are picklable so I'm intrigued to hear this.

Generator functions are generally picklable. E.g.
```python
import dill as pickle

def generator_fn():
    for i in range(10):
        yield i

pickle.dumps(generator_fn)
```

however generators are not picklable
```python
generator = generator_fn()
pickle.dumps(generator)
# TypeError: cannot pickle 'generator' object
```

Though it can happen that some generator functions are not recursively picklable if they use global objects that are not picklable:
```python
def generator_fn_not_picklable():
    for i in generator:
        yield i

pickle.dumps(generator_fn_not_picklable, recurse=True)
# TypeError: cannot pickle 'generator' object
````

I'm trying to create an IterableDataset from a generator but I get this error:
`PicklingError: Can't pickle <built-in function input>: it's not the same object as builtins.input`

What can I do?