Who should I trust: GPT-4 or LLama2-70B?

username: Mengbo_Zhou
cooked: <p>I’d like to know the time ratio of each module during the pre-training process of language models, such as the proportion of Attention. How much impact does it have on the total training time? GPT-4 and LLama2-70B provided me with answers (as shown in the picture). Who should I believe, and is there any professional documentation explaining this?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/6/56abc33e47729083c04616eb182eeb2d7bf07aac.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/56abc33e47729083c04616eb182eeb2d7bf07aac" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/6/56abc33e47729083c04616eb182eeb2d7bf07aac_2_690x426.jpeg" alt="image" data-base62-sha1="cmJ3Rp2gxTv9UXBd0Z8nrpFpYZe" width="690" height="426" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/6/56abc33e47729083c04616eb182eeb2d7bf07aac_2_690x426.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/5/6/56abc33e47729083c04616eb182eeb2d7bf07aac_2_1035x639.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/6/56abc33e47729083c04616eb182eeb2d7bf07aac_2_1380x852.jpeg 2x" data-dominant-color="EAEBEC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1920×1188 252 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>Profile your model and check the real timeline on your system and setup.</p>

username: Mengbo_Zhou
cooked: <p>I may not be able to conduct the experiment due to resource issues, so if there are documentation and links that can help me, I would be extremely grateful !!!</p>

username: J_Johnson
cooked: <p>Most open source LLMs have also published a paper about it. In most cases, the paper describes the resources and training time involved. For example, Llama:</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://arxiv.org/abs/2302.13971">
  <header class="source">
      <img src="https://arxiv.org/static/browse/0.3.4/images/icons/favicon-32x32.png" class="site-icon" width="" height="">

      <a href="https://arxiv.org/abs/2302.13971" target="_blank" rel="noopener nofollow ugc">arXiv.org</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/402;"><img src="https://static.arxiv.org/icons/twitter/arxiv-logo-twitter-square.png" class="thumbnail" width="" height=""></div>

<h3><a href="https://arxiv.org/abs/2302.13971" target="_blank" rel="noopener nofollow ugc">LLaMA: Open and Efficient Foundation Language Models</a></h3>

  <p>We introduce LLaMA, a collection of foundation language models ranging from 7B to 65B parameters. We train our models on trillions of tokens, and show that it is possible to train state-of-the-art models using publicly available datasets exclusively,...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/5/5517357e2e91212cf6bf3f64ac3cb966ca637eef.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/5517357e2e91212cf6bf3f64ac3cb966ca637eef" title="IMG_20231027_113237"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/5517357e2e91212cf6bf3f64ac3cb966ca637eef_2_690x270.jpeg" alt="IMG_20231027_113237" data-base62-sha1="c8Kjdro5VdWXhxglSgIyEVEVuZp" width="690" height="270" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/5517357e2e91212cf6bf3f64ac3cb966ca637eef_2_690x270.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/5517357e2e91212cf6bf3f64ac3cb966ca637eef_2_1035x405.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/5/5517357e2e91212cf6bf3f64ac3cb966ca637eef.jpeg 2x" data-dominant-color="EFEFEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">IMG_20231027_113237</span><span class="informations">1080×424 147 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>With very limited resources, even just CPU ram, you could create a very small version of the model in question, say with only 1 transformer layer, a small embedding layer, perhaps 10,000 tokens wide and 128 elements deep, and then set the model to 4 heads and 128 dim feedforward, and use a loop with timeit or <a href="https://pytorch.org/tutorials/recipes/recipes/benchmark.html" rel="noopener nofollow ugc">torch.utils.benchmark.Timer</a>. This should run relatively quickly.</p>
<p>You can then time test each part of the model and calculate the percentage of time the embedding layer, attention layer, etc. takes from there and extrapolate those values.</p>
<p>Of course, GPUs will be much faster and may vary some in how they approach various matrix operations vs. CPU on the backend. Additionally, various GPUs will vary in processing time, depending on a host of factors, including dtype used, model size, number of cores, throughput, etc.</p>

username: Mengbo_Zhou
cooked: <p>Thank you for your response. I appreciate your feedback.</p>