What is the typical scale of model parameters?

YuA: <p>I think my model’s parameters are very small. The histgrams below shows that most of the parameters are &lt; 0.1. I am worried that this makes my model sensitive to floating point precision.<br>
My model is a very typical autoregressive generative model with 12 decoder-only transformer layers (almost identical to GPT-2)<br>
Is the scale of my model’s parameters acceptable?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/e/5e6f4e6e9cb8e72589a5504b360351219563317c.png" data-download-href="https://discuss.pytorch.org/uploads/default/5e6f4e6e9cb8e72589a5504b360351219563317c" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/e/5e6f4e6e9cb8e72589a5504b360351219563317c_2_690x387.png" alt="image" data-base62-sha1="dtpm0JWuPfQXs9rpdFX84KLsno0" width="690" height="387" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/e/5e6f4e6e9cb8e72589a5504b360351219563317c_2_690x387.png, https://discuss.pytorch.org/uploads/default/original/3X/5/e/5e6f4e6e9cb8e72589a5504b360351219563317c.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/e/5e6f4e6e9cb8e72589a5504b360351219563317c.png 2x" data-dominant-color="ECF3FB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">899×505 20.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>You can create these histgrams with your own model parameters with the code below.<br>
Can anyone share your model’s parameter scale?</p>
<pre><code class="lang-auto">import numpy as np
import torch
import matplotlib.pyplot as plt

def visualize_parameter_scale(state_dict_path:str, n_shown:int=1e5)-&gt;None:
    """Visualize your model's parameter scale with histgrams.
    Args:
        state_dict_path (str): Path to a state dict that stores model parameters.
        n_shown (int, optional): If specified, the given number of parameters are randomly sampled for plot.
            The default is 100,000. If you want to plot everyhting, set this to None.
    Returns:
        None
    """
    # Load state dict
    sd = torch.load(state_dict_path, map_location="cpu")
    # Collect params
    weights = []
    biases = []
    for k, v in sd.items():
        f = v.flatten().numpy()
        if ".weight" in k:
            weights.append(f)
        elif ".bias" in k:
            biases.append(f)
    weights = np.hstack(weights)
    biases = np.hstack(biases)
    if n_shown is not None:
        n_shown = int(n_shown)
        weights = np.random.choice(weights, n_shown)
        biases = np.random.choice(biases, n_shown)
    # plot
    fig, axes = plt.subplots(1, 2, figsize=(10, 5))
    axes[0].hist(x=weights, bins=np.logspace(-5, 1, 100), color='dodgerblue', alpha=0.75)
    axes[0].set_xscale('log')
    axes[0].set_title("Weights")
    axes[1].hist(x=biases, bins=np.logspace(-5, 1, 100), color='dodgerblue', alpha=0.75)
    axes[1].set_xscale('log')
    axes[1].set_title("Biases")
    fig.supxlabel('Element value')
    fig.supylabel('Number of weight elements')
    fig.suptitle("Model parameter element scale")
    plt.show()
</code></pre>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/yua">@YuA</a>,</p>
<p>Floating point round error shouldn’t be an issue at 1e-4, and are common around 1e-7 (for single precision).</p>
<p>Also, a more efficient method to get all the parameters can be found below,</p>
<pre><code class="lang-auto">import torch
from torchvision import models

net = models.resnet18() #example model

params = torch.cat([p.view(-1) for p in net.parameters()], dim=0)

print(params.shape) #returns torch.Size([11689512])
</code></pre>

KFrank: <p>Hi Alpha (and Yu)!</p>
<aside class="quote no-group" data-username="AlphaBetaGamma96" data-post="2" data-topic="207079" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/3da27b/48.png" class="avatar"> AlphaBetaGamma96:</div>
<blockquote>
<p>Floating point round error shouldn’t be an issue at 1e-4, and are common around 1e-7 (for single precision).</p>
</blockquote>
</aside>
<p>This isn’t really right.  Floating-point numbers with smaller magnitudes<br>
aren’t any more subject to round-off error.  (A single-precision float<br>
does start to <em>underflow</em> at around <code>1.e-38</code>, but this is underflow,<br>
rather than round-off error.)</p>
<p>If you did successful computations with a bunch of floating-point<br>
numbers all with magnitude of order one, the same computations<br>
would work just fine if you were to first multiply all the numbers by,<br>
say, <code>1.e-20</code>.  (In some sense, this is the whole point of floating-point<br>
arithmetic.)</p>
<p>Best.</p>
<p>K. Frank</p>

YuA: <p>You made it very clear for me. Thank you very much Frank.<br>
I am now sure that the parameter scale of my model is not a problem.<br>
However, I still cannot solve the problem that I am facing, apparently related to the numerical accuracy. I created a new topic:</p><aside class="quote quote-modified" data-post="1" data-topic="207233">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yua/48/71718_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/outputs-are-different-depending-on-the-input-size/207233">Outputs are different depending on the input size</a> 
  </div>
  <blockquote>
    Thank you for reading my post. 
My question is, how can I mitigate the small numerical errors results from input size variety to accumulate through the layers? 
This question may not be obvious. So I will explain it below with a figure illustrating this problem. 
Backgroud: 
Because of the limit of numerical accuracy, the input size (shape) influences the outputs even though the expected outputs are mathmatically identical. This is explained in the doc <a href="https://pytorch.org/docs/stable/notes/numerical_accuracy.html" class="inline-onebox" rel="noopener nofollow ugc">Numerical accuracy — PyTorch 2.4 documentat…</a>
  </blockquote>
</aside>

<p>I will appreciate it very much if you can give me more insights into this problem.<br>
Thank you very much again for your help.</p>

YuA: <p>Thank you for your answer!</p>