Pytorch load and train data async

Alfons0329: <p>Hello, it is my first time in the forum<br>
Recently I am doing a medical image research, which requires loading a very big pathology slide(about 5GBytes), and doing some CNN jobs.</p>
<h2><a name="problem-1" class="anchor" href="#problem-1"></a>Problem</h2>
<p>Is it possible to <strong>run dataloading and model.run() in an async fashion (producer and consumer)</strong>?</p>
<ul>
<li>model.run() is executing in GPU</li>
<li>dataloading() is executing in CPU<br>
I have understand the ideas and usage of python asyncio library<br>
And the toy example works fine below</li>
</ul>
<h2><a name="toy-example-code-and-profiling-result-2" class="anchor" href="#toy-example-code-and-profiling-result-2"></a>Toy example code and profiling result</h2>
<ul>
<li>Code<br>
(reason I set wait_consumer to 3.2 and wait_producer to 1.7 is because that in the real scenario, the consumer (<code>model.run()</code> takes 320 sec to finish while 170 for <code>imgloader</code> )</li>
</ul>
<pre data-code-wrap="py"><code class="lang-py">wait_consumer = 3.2
wait_pdocuer = 1.7
async def consumer(q):
    while True:
        x, xx = await q.get()
        q.task_done()


async def producer(q, n):
    for x in range(n):
        await asyncio.sleep(wait_producer) # IO takes time for 10 sec
        await q.put((x, x * x))

async def async_main(n):
    q = asyncio.Queue()
    consumers = asyncio.ensure_future(consumer(q))
    await producer(q, n)
    await q.join()
    consumers.cancel() 

def main_run_async(n):
    t_start = time.time()

    loop = asyncio.get_event_loop()
    loop.run_until_complete(async_main(n))
    loop.close()

    t_end = time.time()
    return t_end - t_start
</code></pre>
<ul>
<li>Profiling the results<br>
The async property works well, the orange and blue bar symbolizes the async, producer and consumer version while the green bar is the naïve serial one.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/9/997623cc51388515e8647d38c7c95b48ebdf551b.png" data-download-href="https://discuss.pytorch.org/uploads/default/997623cc51388515e8647d38c7c95b48ebdf551b" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/9/997623cc51388515e8647d38c7c95b48ebdf551b_2_623x500.png" alt="image" data-base62-sha1="lTAajGm3VPIHHFqZzqJz9nQBS9t" width="623" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/9/997623cc51388515e8647d38c7c95b48ebdf551b_2_623x500.png, https://discuss.pytorch.org/uploads/default/original/3X/9/9/997623cc51388515e8647d38c7c95b48ebdf551b.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/9/997623cc51388515e8647d38c7c95b48ebdf551b.png 2x" data-dominant-color="D4DDD7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">870×698 33.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></li>
</ul>
<h2><a name="project-code-3" class="anchor" href="#project-code-3"></a>Project code</h2>
<ul>
<li>Now change the asyncio.sleep() to real code in my research project (some of the name has been hidden due to confidentiality, but most of the architecture works fine and very similar to that of above)</li>
</ul>
<pre data-code-wrap="py"><code class="lang-py">async def model_run_async(model):
    model.run() # Model run in GPU


async def fun_consumer(q, args):
    while True:
        iteration, img_loader = await q.get() # Popped the produced img_loader from queue

        model = ProjectModel(
            image_loader=img_loader,
            image_size=args.img_size,
        )

        print('[ASYNC LOG CONS] Start time {}'.format(time.time() - t_zero))
        t_consumer_start.append(time.time() - t_zero)
        await model_run_async(model)
        t_consumer_end.append(time.time() - t_zero)
        print('[ASYNC LOG CONS] End time {}'.format(time.time() - t_zero))
        q.task_done()

async def image_load_async(args):
    img_loader = BaselineImageLoader(
            slide_path=args.slide_path,
            resize_ratio=args.resize_ratio,
            img_size=args.img_size,
    )
    return img_loader

async def fun_producer(q, args):
    for iteration in range(args.iters):

        print('[ASYNC LOG PROD] Start time {}'.format(time.time() - t_zero))
        t_producer_start.append(time.time() - t_zero)
        img_loader = await image_load_async(args) # block here and hope I can run consumer in this stage (train the last-batch data during loading the current batch of data)
# Load big medical slide data in CPU
        print('[ASYNC LOG PROD] End time {}'.format(time.time() - t_zero))
        q.put((iteration, img_loader)) # Put into queue
        t_producer_end.append(time.time() - t_zero)

async def async_main(args):
    q = asyncio.Queue()
    consumers = asyncio.ensure_future(fun_consumer(q, args))
    await fun_producer(q, args)
    await q.join()
    consumers.cancel()

def main_run_async(args):
    t_start = time.time()

    loop = asyncio.get_event_loop()
    loop.run_until_complete(async_main(args))
    loop.close()

    t_end = time.time()
    return t_end - t_start
</code></pre>
<ul>
<li>Profiling the results<br>
Only the producer seems to be executing while the consumer does not work similar to that of the toy example (Sorry that I cannot post more than 1 image)<br>
[ASYNC LOG PROD] Start time 0, End time 166<br>
[ASYNC LOG PROD] Start time 166, End time 332<br>
[ASYNC LOG PROD] Start time 332, End time 499<br>
… etc, but <strong>WITHOUT THE LOG FROM CONSUMER (model run)</strong></li>
</ul>
<p>Thank you very much in advance for the kindness help, it really helps my research.</p>

Alfons0329: <p>My leased GPU server HW and SW configuration</p>
<pre><code class="lang-auto">CPU: Xeon Gold 6154
GPU: NVIDIA Tesla V100

Ubuntu 18.04.5 LTS
NVIDIA-SMI 450.80.02 CUDA 11.1

Pytorch: 1.8.0a0+17f8c32
Torchvision: 0.8.0a0
Python: 3.6.10 (anaconda, unable to change since I am constrained in container)
</code></pre>

ptrblck: <p>Would it be possible to write a custom <code>Dataset</code> as described <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html">here</a> and pass it to a <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader"><code>DataLoader</code></a>, which can use multiple workers and could thus hide the latency of the data loading and processing?</p>