Random seed closeness

username: Introvert_Loner
cooked: <p>I use these 3 `set_determinism(seed);torch.manual_seed(seed);random.seed(seed)` to set seed for training Pytorch neural networks, does closeness in the choice of seed value (e.g. 10,11,12) have any differences than choosing seed values that are way different from each other (e.g. 1,101, 9999)?</p>

username: Arunprakash-A
cooked: <aside class="quote no-group" data-username="Introvert_Loner" data-post="1" data-topic="222805">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/i/858c86/48.png" class="avatar"> Introvert_Loner:</div>
<blockquote>
<p>have any differences</p>
</blockquote>
</aside>
<p>Could you elaborate on that? A difference in what? Do you mean in terms of the network’s performance?</p>
<p>In general, two random sequences A and B that were generated using two different seeds (regardless of how close the seed values are: for example, 10 and 11 or 10 and 1000) are expected to be uncorrelated and orthogonal in theory. In practice, what we use is called a Pseudo-Random Number Generator; therefore, they will be close to zero correlation and approximately orthogonal, which is suitable for most applications.</p>

username: Introvert_Loner
cooked: <p>I’m sorry for the unclarity, what I mean by “differences” is the loss values of training and validation sets. For example, if I set seed = 42, then I get a validation loss of 0.5, then would setting seed = 43 or 41 make the validation loss more similar to 0.5 than setting seed = 1000? Assume that we only train an simple MLP on MNIST dataset.</p>
<p>P/s: could you please recommend me some papers about the “random seed” in Pytorch or even Python?</p>

username: Arunprakash-A
cooked: <aside class="quote no-group" data-username="Introvert_Loner" data-post="3" data-topic="222805">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/i/858c86/48.png" class="avatar"> Introvert_Loner:</div>
<blockquote>
<p>validation loss more similar to 0.5</p>
</blockquote>
</aside>
<p>If you are using neural networks, there could be many possible reasons for this</p>
<ol>
<li>
<p>Why do we use a random seed?</p>
<p>We need to initialize the neural network weights randomly. A random seed ensures reproducibility by controlling the random number generator, which typically samples from a given distribution (often Gaussian or uniform).</p>
</li>
<li>
<p>How <strong>close (or similar)</strong> are the generated random sequences to each other?</p>
<p>Since the dimension of the generated sequences is very high, traditional distance measures for “closeness” become less meaningful due to the curse of dimensionality.</p>
</li>
<li>
<p>How <strong>close</strong> are the generated random sequences on the loss surface?</p>
<p>This question does not make sense. The random sequence corresponds to a weight vector in weight space, and the loss value does not imply closeness between weight vectors. For example, <strong>two very different weight vectors</strong> can yield the <strong>same</strong> loss of 0.5, especially in flat regions of the loss surface. In essence, it is a many-to-one mapping: many weight configurations can lead to the same loss value.</p>
</li>
</ol>
<aside class="quote no-group" data-username="Introvert_Loner" data-post="3" data-topic="222805">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/i/858c86/48.png" class="avatar"> Introvert_Loner:</div>
<blockquote>
<p>please recommend me some papers about the “random seed”</p>
</blockquote>
</aside>
<p>For that, one needs to study the working principle behind pseudo-random number generators. However, my knowledge in this area is limited.</p>

username: Introvert_Loner
cooked: <p>If the seeds are “close” in my question above, the initialization of weights &amp; biases of one seed should be close to another in the high dimensional space (or you can project N-dimensional flattened vector of weights&amp;biases into 2d space using UMAP) if the 2 seeds are close (e.g. 1000 and 1001). Then, if I am to re-elaborate your answer: the closeness of the seeds have nothing to do with the position of the initialization of weights&amp;biases, does that what you mean?</p>

username: Arunprakash-A
cooked: <aside class="quote no-group" data-username="Introvert_Loner" data-post="5" data-topic="222805">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/i/858c86/48.png" class="avatar"> Introvert_Loner:</div>
<blockquote>
<p>nothing to do</p>
</blockquote>
</aside>
<p>Yes, that’s correct. The closeness of two seeds (e.g., 1000 and 1001) has no relation to the closeness of their initialised weights and biases (especially, in the high-dimensional space). As far as I know, there’s no principle guaranteeing that nearby seeds produce similar starting points (in the loss space)</p>