How partially load big tensor from file?

kenenbek: <p>I can load a tensor from file like this:<br>
<code>X = torch.load(filename)</code><br>
This tensor has a shape <code>torch.Size([30000000])</code>.</p>
<p>But I can’t load dataset fully due to memory limits.</p>
<p>How can I load only first 3000 numbers from file?<br>
And then second portion of 3000 numbers without full load?</p>

Scott_Hoang: <p>I guess an easy solution is to physically partition your file.<br>
another options is to read the file as incoming byte streams, stop reading when you reach desired size, and resume reading once you need more data.</p>

kenenbek: <p>Thank you, it’s a good idea.<br>
How can I transform a byte stream to pytorch tensor?</p>

Scott_Hoang: <p>the tricky part about reading your data as a byte stream is that there is no data structure.<br>
let say x = (BxCxWxH), then your bytes stream has a size of  x = (B * C *W * H). Then you must know the type of your data, i.e int, float, double …etc, and figure out of many bytes each type are to read correctly.<br>
here is some code to get you started</p>
<pre><code class="lang-auto">f = open(&lt;your file path&gt;, 'rb') #  rb is readbytes.
# let assume your matrix is int_8. i.e 8 bits or 1 bytes per int
# loading one Batch
batch = np.frombuffer(f.read(1 * C*W*H), dtype=np.uint8)
batch = batch.reshape((C,W,H))
tensor_batch = torch.from_numpy(batch)
f.close() 
# Im writing this code on the fly. might have some bugs... but you should get the idea.
</code></pre>

kenenbek: <p>I asked here<br>
</p><aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/62600831/how-to-read-longtensor-stored-on-a-disk-by-chunks" target="_blank" rel="nofollow noopener">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/5537840/kenenbek-arzymatov" target="_blank" rel="nofollow noopener">
    <img alt="Kenenbek Arzymatov" src="https://i.stack.imgur.com/aKRoy.jpg?s=128&amp;g=1" class="thumbnail onebox-avatar" width="60" height="60">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/62600831/how-to-read-longtensor-stored-on-a-disk-by-chunks" target="_blank" rel="nofollow noopener">How to read LongTensor stored on a disk by chunks</a>
</h4>

<div class="tags">
  <strong>python, numpy, file, buffer</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/5537840/kenenbek-arzymatov" target="_blank" rel="nofollow noopener">
    Kenenbek Arzymatov
  </a>
  on <a href="https://stackoverflow.com/questions/62600831/how-to-read-longtensor-stored-on-a-disk-by-chunks" target="_blank" rel="nofollow noopener">06:28PM - 26 Jun 20 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


Scott_Hoang: <p>make sure that there is no meta data saved before the actual data.<br>
also:<br>
f = open(filepath, ‘rb’)</p>

kenenbek: <aside class="quote no-group" data-username="Scott_Hoang" data-post="6" data-topic="86088">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/scott_hoang/48/9907_2.png" class="avatar"> Scott_Hoang:</div>
<blockquote>
<p>rb</p>
</blockquote>
</aside>
<p>You are right, I missed ‘rb’ flag.<br>
Do you know what type numpy type correspond to LongTensor?</p>

Scott_Hoang: <p>it should be np.int64</p>

kenenbek: <pre><code># read data
x = np.arange(10, dtype=np.int64)
np.save("1.npy", x)

# open data
f = open("1.npy", "rb")
print(np.frombuffer(f.read(8), dtype=np.int64))
f.close()

print(np.load("1.npy"))
</code></pre>
<p>Gives some wrong number.</p>
<p>But np.load(“1.npy”) works as expected.</p>

Scott_Hoang: <p>The reason why is because np.save stored meta data before the actual data.<br>
I think this is a more elegant solution for your problem<br>
</p><aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/42727412/efficient-way-to-partially-read-large-numpy-file" target="_blank" rel="nofollow noopener">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/5080995/martianwars" target="_blank" rel="nofollow noopener">
    <img alt="martianwars" src="https://graph.facebook.com/10153059476487462/picture?type=large" class="thumbnail onebox-avatar" width="60" height="60">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/42727412/efficient-way-to-partially-read-large-numpy-file" target="_blank" rel="nofollow noopener">Efficient way to partially read large numpy file?</a>
</h4>

<div class="tags">
  <strong>python, numpy</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/5080995/martianwars" target="_blank" rel="nofollow noopener">
    martianwars
  </a>
  on <a href="https://stackoverflow.com/questions/42727412/efficient-way-to-partially-read-large-numpy-file" target="_blank" rel="nofollow noopener">08:44PM - 10 Mar 17 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>