Saving Graph/Structured Data in Datasets
State: closed | Author: gsh199449 (NONE)
Labels: name: enhancement

Thanks for this amazing library! And my question is I have structured data that is organized with a graph. For example, a dataset with users' friendship relations and user's articles. When I try to save a python dict in the dataset, an error occurred ``did not recognize Python value type when inferring an Arrow data type''.
Although I also know that storing a python dict in pyarrow datasets is not the best practice, but I have no idea about how to save structured data in the Datasets. 

Thank you very much for your help.

Comments: It should probably work out of the box to save structured data. If you want to show an example we can help you.

An example of a toy dataset is like:
```json
[
    {
        "name": "mike",
        "friends": [
            "tom",
            "lily"
        ],
        "articles": [
            {
                "title": "aaaaa",
                "reader": [
                    "tom",
                    "lucy"
                ]
            }
        ]
    },
    {
        "name": "tom",
        "friends": [
            "mike",
            "bbb"
        ],
        "articles": [
            {
                "title": "xxxxx",
                "reader": [
                    "tom",
                    "qqqq"
                ]
            }
        ]
    }
]
```
We can use the friendship relation to build a directional graph, and a user node can be represented using the articles written by himself. And the relationship between articles can be built when the article has read by the same user.
This dataset can be used to model the heterogeneous relationship between users and articles, and this graph can be used to build recommendation systems to recommend articles to the user, or potential friends to the user.

Hi,

you can do the following to load this data into a `Dataset`:
```python
from datasets import Dataset
examples = [
    {
        "name": "mike",
        "friends": [
            "tom",
            "lily"
        ],
        "articles": [
            {
                "title": "aaaaa",
                "reader": [
                    "tom",
                    "lucy"
                ]
            }
        ]
    },
    {
        "name": "tom",
        "friends": [
            "mike",
            "bbb"
        ],
        "articles": [
            {
                "title": "xxxxx",
                "reader": [
                    "tom",
                    "qqqq"
                ]
            }
        ]
    }
]

keys = examples[0].keys()
values = [ex.values() for ex in examples]
dataset = Dataset.from_dict({k: list(v) for k, v in zip(keys, zip(*values))})
```

Let us know if this works for you.

Thank you so much, and that works! I also have a question that if the dataset is very large, that cannot be loaded into the memory. How to create the Dataset?

If your dataset doesn't fit in memory, store it in a local file and load it from there. Check out [this chapter](https://huggingface.co/docs/datasets/master/loading_datasets.html#from-local-files) in the docs for more info.

Nice! Thanks for your help.