How accurate are RNNs for just predicting numerical values like x,y coordinates etc.?

zimmer550: <p>So, for example, if I just feed x, y coordinates at consecutive timesteps to an encoder and try to predict future x, y coordinates using a decoder, how confident can I be of the learning ability of this RNN? Also, to aid the RNN, let’s say I add some supporting information but I only add it in the decoder stage and not the encoder stage, then how good can my RNN be at learning good weights to do a prediction of x, y coordinates?</p>
<p>The reason I am asking this is because I see a lot of papers where, to predict spatial coordinates, the authors just feed raw x, y coordinates to the RNN encoder. But I think that may be a bit naive even if the encoded representation is supported by additional information (like some features extracted from the images) because just feeding a bunch of raw numbers at the encoder stage without any other additional ‘contextual’ information might mean that the encoder just learns something trivial like a constant-velocity Kalman filter (even after backprop from later on where more information is added in).</p>
<p>For example, the image is from the paper: <a href="https://arxiv.org/pdf/1809.07408.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1809.07408.pdf</a><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/9/89bb9db6b6bfae7aadda5373c3366ddf1291096a.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/89bb9db6b6bfae7aadda5373c3366ddf1291096a" title="PNG"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/9/89bb9db6b6bfae7aadda5373c3366ddf1291096a_2_690x250.jpeg" alt="PNG" data-base62-sha1="jErnuYvV3IqPEcza8Ek6Ot4J3zI" width="690" height="250" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/9/89bb9db6b6bfae7aadda5373c3366ddf1291096a_2_690x250.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/8/9/89bb9db6b6bfae7aadda5373c3366ddf1291096a_2_1035x375.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/9/89bb9db6b6bfae7aadda5373c3366ddf1291096a_2_1380x500.jpeg 2x" data-dominant-color="E3E6E2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">PNG</span><span class="informations">1552×563 159 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Look at the part of Location-Scale Encoder where they just feed the bounding box coordinates at each time step of the encoder. Isn’t it a bad idea to just feed numerical values like this with no other contextual information? I mean, even though they later combine the encoded representation with image information but I think this part of the architecture might not get trained appropriately.</p>
<p>However, if a lot of researchers seem to be doing this then that means I am not seeing something. What do you guys think?</p>

tom: <p>Well, the charm of neural networks is that you’re basically calculating quite complex functions with mostly linear + pointwise nonlinearities.</p>
<p>In a bit more detail: Feeding the coordinates often, but not always, works well. For example, the “positional encoding” bit in the famous <em>Attention is all you need</em> paper introducing transformers is a case when people chose to <em>not</em> just put in the index.</p>
<p>So the gist here is: try simple, be fancy if it doesn’t work.</p>
<p>Another tangentially related occasion where just position doesn’t work is the loss function, where one might naively use mean squared error, but doesn’t whenever one has a credible alternative (like cross-entropy).</p>
<p>Best regards</p>
<p>Thomas</p>

zimmer550: <blockquote>
<p>Another tangentially related occasion where just position doesn’t work is the loss function, where one might naively use mean squared error, but doesn’t whenever one has a credible alternative (like cross-entropy).</p>
</blockquote>
<p>mean squared loss, L2 loss and L1 loss are most suited for regression right? Why would you want to use cross-entropy which is otherwise used for binary classification?</p>

tom: <p>Yes, but so the observation is that the regressing on the class number doesn’t work, even if the perfect classifier would allow to make it this formulation with error 0.</p>
<p>Best regards</p>
<p>Thomas</p>