PyTorch Geometric -- how make own dataset of multiple graphs?

David_Ireland: <p>I have a list of multiple Data objects that are all independent graphs. How do I make a dataset out of this, so that it is like the built in datasets in <a href="https://pytorch-geometric.readthedocs.io/en/latest/notes/introduction.html#common-benchmark-datasets" rel="noopener nofollow ugc">the tutorial here</a>? I have tried the <a href="https://pytorch-geometric.readthedocs.io/en/latest/notes/create_dataset.html#creating-in-memory-datasets" rel="noopener nofollow ugc">tutorial on making your own dataset</a> but I have absolutely no idea how to make sense of it (note I am experienced with PyTorch but not so much with custom data sets, usually they are not needed).</p>
<p>All I want to do is make this list of Data objects a dataset so that it has the same functionality as in the tutorial but this seems impossible to do – if someone could help I would be extremely grateful, thanks.</p>

Anil_Kamat: <p>I am also looking for this answer. Most of the online sources only talk about the in-built dataset but to apply GNN we need to synthesize our own data. if you have found the answer do share. Thanks.<br>
I think the following method from the PyG doc will work but it is not an efficient way to do::</p>
<pre><code class="lang-auto">from torch_geometric.data import Data
from torch_geometric.loader import DataLoader

data_list = [Data(...), ..., Data(...)]
loader = DataLoader(data_list, batch_size=32)

</code></pre>
<p>Here, each Data object should be created in a loop.</p>

Animesh_Basak_Chowdh: <p>Checkout these two files:</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/NYU-MLDA/OpenABC/blob/master/datagen/utilities/PyGDataAIG.py">
  <header class="source">

      <a href="https://github.com/NYU-MLDA/OpenABC/blob/master/datagen/utilities/PyGDataAIG.py" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/NYU-MLDA/OpenABC/blob/master/datagen/utilities/PyGDataAIG.py" target="_blank" rel="noopener nofollow ugc">NYU-MLDA/OpenABC/blob/master/datagen/utilities/PyGDataAIG.py</a></h4>


      <pre><code class="lang-py">import os
import shutil

import pandas as pd
import networkx as nx
import glob
import pickle
import copy

from typing import Optional, Tuple

import torch
from torch import Tensor
from torch.utils.dlpack import to_dlpack, from_dlpack
import scipy.sparse
import zipfile
import argparse

import torch_geometric
import torch_geometric.data
</code></pre>



  This file has been truncated. <a href="https://github.com/NYU-MLDA/OpenABC/blob/master/datagen/utilities/PyGDataAIG.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<aside class="onebox githubblob" data-onebox-src="https://github.com/NYU-MLDA/OpenABC/blob/master/models/qor/SynthNetV1/netlistDataset.py">
  <header class="source">

      <a href="https://github.com/NYU-MLDA/OpenABC/blob/master/models/qor/SynthNetV1/netlistDataset.py" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/NYU-MLDA/OpenABC/blob/master/models/qor/SynthNetV1/netlistDataset.py" target="_blank" rel="noopener nofollow ugc">NYU-MLDA/OpenABC/blob/master/models/qor/SynthNetV1/netlistDataset.py</a></h4>


      <pre><code class="lang-py">import os.path as osp
import torch
from zipfile import ZipFile

import pandas as pd
from torch_geometric.data import Dataset, download_url


class NetlistGraphDataset(Dataset):
    def __init__(self, root, filePath, transform=None, pre_transform=None):
        self.filePath = osp.join(root, filePath)
        super(NetlistGraphDataset, self).__init__(root, transform, pre_transform)

    @property
    def processed_file_names(self):
        fileDF = pd.read_csv(self.filePath)
        return fileDF['fileName'].tolist()

    def len(self):
        return len(self.processed_file_names)
</code></pre>



  This file has been truncated. <a href="https://github.com/NYU-MLDA/OpenABC/blob/master/models/qor/SynthNetV1/netlistDataset.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>I have created first done the preprocessing and stored .pt files in processed directory. Later, I’m loading it via NetlistGraphDataset. It is flexible if one wants to create different train/test split on different graph objects.</p>