What memory is used when downloading data on each rank - multiple GPUs

PopHSKE: <p>I have I am sure a basic question / misunderstanding. I am training a model using DDP on AWS Sagemaker. I am using a single instance with multiple GPUs. There is a large dataset on S3 and it seems I need to download the data on each of the ranks. The data is downloaded and then a dataset/data loader is created. If data is downloaded - read into memory via pandas - does that mean that the GPU memory is being used to read in the data into pandas since it needs to happen on each rank?</p>

ptrblck: <p>Typical DDP setups use a <code>DistributedSampler</code> to avoid reusing the same samples and to split them among the ranks. Are you using such a sampler already?</p>

PopHSKE: <p>I am. To give a little more detail, I am essentially following tutorials such as <a href="https://github.com/pytorch/examples/blob/main/distributed/ddp-tutorial-series/multigpu_torchrun.py" class="inline-onebox" rel="noopener nofollow ugc">examples/distributed/ddp-tutorial-series/multigpu_torchrun.py at main · pytorch/examples · GitHub</a>. (Imagine in this script instead of generating the random data, it is reading a text file from S3.)</p>
<p>where for each rank (GPU when using a single instance/node with multiple GPUs) the same dataset appears to be read into memory. So, in my script I have a step that reads data from S3 into a pandas dataframe. It seems I need to do this on each rank. When doing this, is the process loading the data into GPU memory?</p>

ptrblck: <p>I assume you are loading the entire dataset into the host RAM on each rank which is a bit wasteful. A common approach would be to use a lazy loading approach. In this case you would load the dataset location or sample paths in the <code>Dataset.__init__</code> method and would load the needed sample in the <code>Dataset.__getitem__</code>. The sampler would then make sure to only load the needed samples on each rank.</p>

PopHSKE: <p>Yes I think I am. In a script using DDP, would this mean that the data is being loaded into GPU RAM?</p>
<p>Dataset.<strong>getitem</strong> reads from disk, is that would you mean?</p>

ptrblck: <aside class="quote no-group" data-username="PopHSKE" data-post="5" data-topic="197125">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/a9adbd/48.png" class="avatar"> PopHSKE:</div>
<blockquote>
<p>would this mean that the data is being loaded into GPU RAM?</p>
</blockquote>
</aside>
<p>Not necessarily as it depends on the <code>Dataset.__getitem__</code> implementation and if you would move the data to the device (which is not the standard use case as the data is loaded on the host in the <code>Dataset</code> and moved inside the training loop to the GPU).</p>

PopHSKE: <p>So when a dataset (e.g. csv read into pandas) occurs in a script using DDP as in the tutorial above ( <a href="https://github.com/pytorch/examples/blob/main/distributed/ddp-tutorial-series/multigpu_torchrun.py" class="inline-onebox" rel="noopener nofollow ugc">examples/distributed/ddp-tutorial-series/multigpu_torchrun.py at main · pytorch/examples · GitHub</a>) on a single instance with multiple GPUs like this:</p>
<p>df = pd.read_csv(‘s3://location_of_file’)<br>
and then its loaded like into pytorch like:</p>
<p>X = torch.tensor(df.values, dtype=torch.float32)<br>
loader = DataLoader(X)</p>
<p>this occurs in some non GPU memory that is somehow associated with each process (rank)?</p>

ptrblck: <aside class="quote no-group" data-username="PopHSKE" data-post="7" data-topic="197125">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/a9adbd/48.png" class="avatar"> PopHSKE:</div>
<blockquote>
<p>this occurs in some non GPU memory</p>
</blockquote>
</aside>
<p>Yes, in host RAM.</p>
<aside class="quote no-group" data-username="PopHSKE" data-post="7" data-topic="197125">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/a9adbd/48.png" class="avatar"> PopHSKE:</div>
<blockquote>
<p>that is somehow associated with each process (rank)?</p>
</blockquote>
</aside>
<p>It’s just the RAM available in your system that can be used by all processes.</p>