How to use Torchdata to load tar files from s3 so that the datapipe return `(sampel, label)`

knoriy: <p>I’m trying to learn more about torchdata datapipes.</p>
<p>I’m trying to load a <code>.tar</code> file from s3, which contains wavs and text labels in the form of <code>wav</code> and <code>json</code> files. I hope to end up with a <code>dataset</code> that can be passed to <code>DataLoader(...)</code>.</p>
<p>So far, I have:</p>
<p>The code below takes an s3 dir and lists all files in the bucket, for example, returning <code>s3://my_bucket/0.tar</code></p>
<pre><code class="lang-auto">dp_s3_urls = td.datapipes.iter.IterableWrapper(["s3://my_bucket/"])\
        .list_files_by_s3()\
        .shuffle()\
        .sharding_filter()
</code></pre>
<p>Load the tar file:</p>
<pre><code class="lang-auto">td.datapipes.iter.S3FileLoader(dp_s3_urls).load_from_tar()
</code></pre>
<p>which returns:</p>
<pre><code class="lang-auto">('s3://my_bucket/0.tar/data/1.json', StreamWrapper&lt;s3:/my_bucket/0.tar/data/1.json,&lt;ExFileObject name=None&gt;&gt;)
('s3://my_bucket/0.tar/data/1.wav', StreamWrapper&lt;s3:/my_bucket/0.tar/data/1.wav,&lt;ExFileObject name=None&gt;&gt;)
</code></pre>
<p>I’m stuck with loading this information into a tensor using TorchAudio etc.</p>

nivek: <p>If your tar file is structured in a way such that you always get “1.json” before “1.wav”, you can do <code>dp.batch(2)</code>, which will give you <code>[(s3_path_json, JsonFileStream), (s3_path_wav, WavFileStream)]</code>. You can write your own map function and pass it into <code>dp.map(fn)</code> to handle those afterward as you see fit.</p>
<p>If the order is not guaranteed, you can use <code>dp.demux(2, _fn)</code> to separate them into JSON and .wav files (and process them separately if you like). Then use one of the <code>zip_with_iter </code> to join them back together.</p>

knoriy: <p>Thank you for the help <a class="mention" href="/u/nivek">@nivek</a>.</p>
<p>Here is the code for anyone interested:</p>
<pre><code class="lang-python">import io
import json
import torchdata
import soundfile

def to_sampels(data):
		a, t = data
		return soundfile.read(io.BytesIO(a[1].read())), json.loads(t[1].read().decode('utf-8'))

dp_s3_urls = torchdata.datapipes.iter.IterableWrapper(["s3://m_bucket/"])\
			.list_files_by_s3()\
			.shuffle()\
			.sharding_filter()

datapipe = torchdata.datapipes.iter.S3FileLoader(dp_s3_urls)\
			.load_from_tar() \
			.batch(2) \
			.map(to_sampels)

for i in dp_s3_urls:
    print(i)
</code></pre>
<p>In this example:</p>
<p><code>dp_s3_urls</code> take a <code>s3</code> directory find all file in the dir, and outputs a shuffled list of s3 directories (e.g. [“s3://m_bucket/0.tar”, “s3://m_bucket/1.tar”, …, “s3://m_bucket/n.tar”]).</p>
<p><code>datapipe</code> takes these URLs and loads them into memory, the batch function as mentioned by <a class="mention" href="/u/nivek">@nivek</a> combines the <code>(sample, labels)</code>. You would also use the following to achieve the same result to group your <code>(sample, labels)</code>:</p>
<pre><code class="lang-python">datapipe.groupby(lambda x: os.path.basename(x[0]).split(".")[0],group_size=2, guaranteed_group_size=2)
</code></pre>
<p>.</p>
<p><code>to_sample</code> takes the given streamed data, reads the data and loads the audio using <code>soundfile</code>. The JSON is loaded as a byte string, so this is decoded using <code>utf-8</code> and finally loaded via the <code>json</code> library.</p>
<p>As mentioned in this <a href="https://discuss.pytorh.org/t/how-to-debug-s3-errors-with-torchdata/164254/2?u=knoriy" rel="noopener nofollow ugc">post</a>, you can get:</p>
<pre><code class="lang-auto">ValueError: curlCode: 77, Problem with the SSL CA cert (path? access rights?)
</code></pre>
<p>This was fixed by chaging from <code>S3FileLoader(...)</code> to using <code>FSSpecFileLoader</code></p>
<pre><code class="lang-python">from torch.utils.data import DataLoader

def to_sampels(data):
		a, t = data
		return soundfile.read(io.BytesIO(a[1].read())), json.loads(t[1].read().decode('utf-8'))

datapipe = torchdata.datapipes.iter.IterableWrapper(["s3://m_bucket/"])\
			.list_files_by_fsspec()\
			.shuffle()\
			.sharding_filter()\
			.open_files_by_fsspec(mode='rb')\
			.load_from_tar() \
			.batch(2) \
			.map(to_sampels)

dataloader = DataLoader(dapapipe, batch_size=1, num_workers=6)

for i in dataloader:
    print(i)
</code></pre>

wtliao: <aside class="quote no-group" data-username="knoriy" data-post="3" data-topic="174377">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/knoriy/48/35651_2.png" class="avatar"> knoriy:</div>
<blockquote>
<p><code>torchdata.datapipes.iter.S3FileLoader</code></p>
</blockquote>
</aside>
<p>Thanks for the solution. I have problem when using io.BytesIO(data[0][1]) that TypeError: a bytes-like object is required, not ‘StreamWrapper’. Do you have any idea about that? Thanks!</p>

knoriy: <p>Do you have a minimal code you could share?</p>