How would you suggest storing OpenAI Gym style data?

username: Alio
cooked: <p>If you’re not familiar with an OpenAI Gym it is an agent in a Markov Decision Process that returns data of the observation, the reward, whether the game is over, and some other info. Is there an established / optimal way of dealing with this in LibTorch?<br>
I have a guess at the end of this post, but I’d like to hear any ideas.</p>
<p>I don’t need the C++ version of Gym, which I’m pretty sure doesn’t exist. I’m going to make my own Gym but I don’t know how to store and retrieve this kind of data.</p>
<p>This is typical of what you get back after each ‘step’</p>
<pre><code class="lang-auto">(array([ 0.02957329,  0.20411213, -0.04598045, -0.2613098 ], dtype=float32),
 1.0,
 False,
 {some info})
</code></pre>
<p>This is just what the Gym returns after each step. I need more than this. I need to create large buffers that hold the current state, action, reward, done, and next state.</p>
<p>I have NumCpp (a C++ implementation of Numpy), but I don’t think NumCpp and Libtorch work together like PyTorch and Numpy do on Python. What would I include to work with?</p>
<p>My solution might be to just make a large tensor for everything. A large, long tensor for each state, new state, action, reward, and dones.<br>
i.e. the new states part of the buffer will be a large tensor and I will add each new new state with <code>torch::cat</code> and call all five tensors with the same index number when recalling them or deleting them. I think there’s no way for this to be slow. What do you think?</p>

username: ptrblck
cooked: <p>Based on the current output it seems you are dealing with numpy arrays, floats, bools, and some other objects. I don’t know how you are implementing your custom Gym in C++, but I assume you can deal with tensors. If so, then storing these tensors should work. Concatenating them via <code>torch::cat</code> sounds also reasonable. However,</p>
<aside class="quote no-group" data-username="Alio" data-post="1" data-topic="159719">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alio/48/8142_2.png" class="avatar"> Alio:</div>
<blockquote>
<p>I think there’s no way for this to be slow</p>
</blockquote>
</aside>
<p>this might not be entirely true. Concatenating tensors and increasing their size would need to re-allocate new memory in each iteration:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.randn(10)
for _ in range(10):
    x = torch.cat((x, torch.randn(10), dim=0))
</code></pre>
<p>The better approach might be to store the new tensors in e.g. a <code>list</code> (in Python) or a <code>std::vector</code> (in C++) and create the final tensor afterwards.</p>

username: Alio
cooked: <p>Can only one type go in a std::vector?<br>
I suppose I could just append the reward and the bool as a value in the tensor.</p>

username: ptrblck
cooked: <p>If you plan to create a single output tensor from the content of the <code>std::vector</code> you should not mix the types inside it as it will fail.<br>
Also, make sure the tensors have the right shape so that <code>cat</code> or <code>stack</code> can indeed create the single tensor afterwards.<br>
“Appending” the reward and bool values to a tensor might work in case you could expand the tensor shape and want to slice it after loading.</p>