Host llama2 13b as sagemaker endpoint

Sayli_D: <aside class="onebox githubblob" data-onebox-src="https://github.com/aws/amazon-sagemaker-examples-community/blob/main/torchserve/inf2/llama2/llama-2-13b.ipynb">
  <header class="source">

      <a href="https://github.com/aws/amazon-sagemaker-examples-community/blob/main/torchserve/inf2/llama2/llama-2-13b.ipynb" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/aws/amazon-sagemaker-examples-community/blob/main/torchserve/inf2/llama2/llama-2-13b.ipynb" target="_blank" rel="noopener nofollow ugc">aws/amazon-sagemaker-examples-community/blob/main/torchserve/inf2/llama2/llama-2-13b.ipynb</a></h4>


      <pre><code class="lang-ipynb">{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0524d4b4",
   "metadata": {},
   "source": [
    "#  Using TorchServe on SageMaker Inf2.24xlarge with Llama2-13B"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92a529f6",
   "metadata": {},
   "source": [
    "## Contents\n",
    "\n",
    "This notebook uses SageMaker notebook instance conda_python3 kernel, demonstrates how to use TorchServe to deploy Llama-2-13 on SageMaker inf2.24xlarge. There are multiple advanced features in this example.\n",
    "\n",
    "* Neuronx AOT precompile model\n",
</code></pre>



  This file has been truncated. <a href="https://github.com/aws/amazon-sagemaker-examples-community/blob/main/torchserve/inf2/llama2/llama-2-13b.ipynb" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Could create an endpoint as above for llama 13b base, but it gives a timeout error on container primary for 13b chat.</p>
<p>For above, created the neuron artifacts for the 13b chat model using this -</p><aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/serve/blob/master/examples/large_models/inferentia2/llama2/Readme.md?plain=1#L56">
  <header class="source">

      <a href="https://github.com/pytorch/serve/blob/master/examples/large_models/inferentia2/llama2/Readme.md?plain=1#L56" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/serve/blob/master/examples/large_models/inferentia2/llama2/Readme.md?plain=1#L56" target="_blank" rel="noopener nofollow ugc">pytorch/serve/blob/master/examples/large_models/inferentia2/llama2/Readme.md?plain=1#L56</a></h4>



    <pre class="onebox"><code class="lang-md?plain=1">
      <ol class="start lines" start="46" style="counter-reset: li-counter 45 ;">
          <li>cd serve</li>
          <li></li>
          <li># Install dependencies</li>
          <li>python ts_scripts/install_dependencies.py --neuronx --environment=dev</li>
          <li></li>
          <li># Install torchserve and torch-model-archiver</li>
          <li>python ts_scripts/install_from_src.py</li>
          <li></li>
          <li># Install additional neuron packages, SDK 2.12.2: https://awsdocs-neuron.readthedocs-hosted.com/en/latest/release-notes/prev/content.html#id8</li>
          <li>python -m pip install neuronx-cc==2.8.0.25 torch-neuronx==1.13.1.1.9.1 transformers-neuronx==0.5.58</li>
          <li class="selected"></li>
          <li># Navigate to `examples/large_models/inferentia2/llama2` directory</li>
          <li>cd examples/large_models/inferentia2/llama2/</li>
          <li></li>
          <li># Install additional necessary packages</li>
          <li>python -m pip install -r requirements.txt</li>
          <li>```</li>
          <li></li>
          <li>### Step 3: Save the model artifacts compatible with `transformers-neuronx`</li>
          <li>In order to use the pre-compiled model artifacts, copy them from the model zoo using the command shown below and skip to **Step 5**</li>
          <li>```bash</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
Could start torchserve and run inference via curl command here, so the model artifacts look okay. But the same artifacts won’t work in the first notebook reference link.</p>