DataLoader and Postgres (or other SQL) an option?

JQVeenstra: <p>Hi,</p>
<p>I’m trying to keep things in a postgres database, because - well, it’s complicated.</p>
<p>Is there anyone who’s done this in an efficient manner with the DataLoader and Dataset classes?  I’m relatively proficient at Google-Fu, and no dice so far.</p>
<p>I really would prefer not to have to export from postgres to numpy arrays or csvs, but it seems that those are the best ways I can do this.</p>
<p>Please, someone prove me wrong if you can.</p>
<p>Thanks!</p>
<p>JQ</p>

ptrblck: <p>Would it be possible to query your data using some kind of index? If so you could try to get the sample from your Postgres database in the <code>__getitem__</code> function of your <code>Dataset</code> and try to convert it to a numpy array using e.g. <a href="https://github.com/quantopian/warp_prism">warp_prism</a>.<br>
From there you could transform your numpy array to a tensor using <code>torch.from_numpy</code>.<br>
I haven’t tried it and am absolutely no database expert, but it could be a potential way.<br>
Let me know, if I’m completely missing some crucial aspects.</p>

JQVeenstra: <p>Actually… that might work.  I need to think about it.</p>
<p>No… wait.  It <em>will</em> work.  Thanks.  Brilliant.  I should have thought of that myself.</p>

djseivad: <p>Just thought I would share with ya. Since running so many single record queries can be a bottleneck for performance.</p>
<p>I found considerable speed up when querying my dataset by simply returning my indexes in the getitem function.<br>
Then the list provided to a custom collate function can be converted to a tuple and used with a WHERE IN query which returns the batch records in a single query,</p>

nateraw: <p>I ran into this issue too. I made two dataloaders. First dataloader checks length of table you are querying on <code>__init__</code> and then <code>__getitem__</code> spits out a large amount of row IDs from the database. Second dataloader is initialized with these indices as an <code>__init__</code> parameter, grabs all of those indices from the DB and sets them to <code>self.X</code>. You can then batch off of that second loader to train your model.</p>
<p>Still interested in other solutions, but this solution gets rid of the problem of running select statements on every <code>__getitem__</code> call.</p>
<p>ex. pseudocode:</p>
<pre><code class="lang-python"># Indices is array of row IDs from SQL (mabye 1,000,000 at a time)
for indices in partition_loader:
    # Second loader initialized within this loop w/ those indices
    loader = data.DataLoader(local_dataset(indices, **params) **loader_params)
    for batch_x, batch_y in loader:
        fit_batch(batch_x, batch_y)
</code></pre>

JQVeenstra: <p>I think that this would probably be a cleaner way to do it than the one we settled on.  Possibly more efficient as well.</p>

djseivad: <p>So I obtained batches directly from the psql db with psycopg2 using a single dataloader and dataset.</p>
<p>I was using a connection pool in my instantiation of the dataset object.</p>
<p>I have since switched to fast5 files for my training datasets and using pandas for performance reasons but I prefer the data wrangling capabilities of SQL. Would be nice if training your model this way was more efficient.</p>
<pre><code class="lang-auto">def __getitem__(self, idx: int) -&gt; int:
    return idx

def collate(self, results: list) -&gt; tuple:
    query = """SELECT label, feature FROM dataset WHERE index IN %s"""
    result = None
    conn = self.conn_pool.getconn()

    try:
        conn.set_session(readonly=True)
        cursor = conn.cursor()
        cursor.execute(query, tuple(results,))
        result = cursor.fetchall()

    except Error as conn_pool_error:
        print(conn_pool_error)

    finally:
        self.conn_pool.putconn(conn)
   #format batches here
   #format batches here

    if result is not None:
        return result
    else: 
        throw Exception('problem in obtaining batches')
            

</code></pre>

jwickens: <p>I found it easier to not use Dataset and Dataloader at all and rely on async. Here’s a writeup (with code): <a href="https://medium.com/@jonathan.wickens/using-postgres-as-a-dataloader-with-pytorch-bba0d5cbe1fa" rel="nofollow noopener">https://medium.com/@jonathan.wickens/using-postgres-as-a-dataloader-with-pytorch-bba0d5cbe1fa</a></p>

Cow_woC: <p><a class="mention" href="/u/jwickens">@jwickens</a></p>
<p>Is there a way to wrap this in a DataLoader? PyTorch Lightning requires the use of DataLoaders.</p>

StefanK: <p>Maybe:</p>
<pre><code class="lang-auto">class DBIterDataset (pytorch.IterableDataset):
  def __init__ (self, ait, loop):
    self.ait = ait
    self.loop = loop
  def __iter__ (self):
    return wrap_async_iter (self.ait, self.loop)
</code></pre>