i am trying to follow the steps in the article. i am trying to do it on a windows machine. please assist with the steps: ""Open in app
Sign up

Sign in

Search
Write


Run Ollama Locally Using Google Colab’s Free GPU
Neo@Matrix
Neo@Matrix

·
Follow

2 min read
·
Feb 28, 2024

2



Many computers lack the powerful GPUs required to run large models like Ollama, preventing numerous users from enjoying the conveniences of local large models, such as article optimization, meeting summary extraction, and English email composition. A new method now enables local Ollama invocation of Google Colab’s free GPU for rapid AI response generation.

Step 1: Have a Google Account

Naturally, the first step is to have a Google account. This is widely covered online, so we won’t delve into the details here.

Step 2: Access the Prepared Ollama Notebook

Visit the prepared Ollama.ipynb at https://colab.research.google.com/drive/1JNOrMvmkNvugoglaOKCqceL5XSXCOaAh

Step 3: Register and Get Your Ngrok Token

Sign up for Ngrok (free) and obtain your token at [Ngrok Dashboard](https://dashboard.ngrok.com/get-started/your-authtoken). Fill in your token in the Colab notebook.

Step 4: Enter Your Ngrok Token

In the Colab notebook, replace `token=”Your Ngrok token”` within the code block 3 with your actual Ngrok token.

Step 5: Select the GPU T4 runtime type

Choose the GPU T4 for your session.

Step 6: Execute Steps in the Notebook

Follow the steps 1, 2, 3 in the notebook. After completing step 3, you will receive an URL like `https://xxxxxxx.ngrok-free.app`.

Step 7: Install Ollama on Your Computer

Install Ollama from [Ollama Download Page](https://ollama.com/download), available for macOS, Linux, and Windows.

Step 8: Set Environment Variable

On your computer, set the environment variable with `export OLLAMA_HOST=https://xxxxxxx.ngrok-free.app/`.

Step 9: Run Ollama

Execute `ollama run model_name`, for example, `ollama run gemma`. Wait for the model to load. Although it appears to run locally, it actually invokes the remote Colab’s T4 GPU.

Now, you can input questions to receive answers or use more apps to call Ollama, like setting the OpenAi-Translator tool to use Ollama, bypassing the need for a VPN to use ChatGPT and avoiding account bans.

""
# Download and install ollama to the system
!curl https://ollama.ai/install.sh | sh
[1]
6s
# Download and install ollama to the system
!curl https://ollama.ai/install.sh | sh
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0>>> Downloading ollama...
100 10044    0 10044    0     0  24241      0 --:--:-- --:--:-- --:--:-- 24260
############################################################################################# 100.0%
>>> Installing ollama to /usr/local/bin...
>>> Creating ollama user...
>>> Adding ollama user to video group...
>>> Adding current user to ollama group...
>>> Creating ollama systemd service...
WARNING: Unable to detect NVIDIA/AMD GPU. Install lspci or lshw to automatically detect and install GPU dependencies.
>>> The Ollama API is now available at 127.0.0.1:11434.
>>> Install complete. Run "ollama" from the command line.

!pip install aiohttp pyngrok

import os
import asyncio

# Set LD_LIBRARY_PATH so the system NVIDIA library 
os.environ.update({'LD_LIBRARY_PATH': '/usr/lib64-nvidia'})

async def run_process(cmd):
  print('>>> starting', *cmd)
  p = await asyncio.subprocess.create_subprocess_exec(
      *cmd,
      stdout=asyncio.subprocess.PIPE,
      stderr=asyncio.subprocess.PIPE,
  )

  async def pipe(lines):
    async for line in lines:
      print(line.strip().decode('utf-8'))

  await asyncio.gather(
      pipe(p.stdout),
      pipe(p.stderr),
  )

#register an account at ngrok.com and create an authtoken and place it here
await asyncio.gather(
    run_process(['ngrok', 'config', 'add-authtoken','2e2ndcorRJw6Nd9LPQNpqlzsDWS_8XqdrTZ17vrZtqU3eums'])
)

await asyncio.gather(
    run_process(['ollama', 'serve']),
    run_process(['ngrok', 'http', '--log', 'stderr', '11434']),
)

Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (3.9.3)
Requirement already satisfied: pyngrok in /usr/local/lib/python3.10/dist-packages (7.1.5)
Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp) (1.3.1)
Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp) (23.2.0)
Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp) (1.4.1)
Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp) (6.0.5)
Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp) (1.9.4)
Requirement already satisfied: async-timeout<5.0,>=4.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp) (4.0.3)
Requirement already satisfied: PyYAML>=5.1 in /usr/local/lib/python3.10/dist-packages (from pyngrok) (6.0.1)
Requirement already satisfied: idna>=2.0 in /usr/local/lib/python3.10/dist-packages (from yarl<2.0,>=1.0->aiohttp) (3.6)
>>> starting ngrok config add-authtoken 2e2ndcorRJw6Nd9LPQNpqlzsDWS_8XqdrTZ17vrZtqU3eums
Authtoken saved to configuration file: /root/.config/ngrok/ngrok.yml
>>> starting ollama serve
>>> starting ngrok http --log stderr 11434
time=2024-03-22T13:34:58.599Z level=INFO source=images.go:806 msg="total blobs: 0"
time=2024-03-22T13:34:58.600Z level=INFO source=images.go:813 msg="total unused blobs removed: 0"
time=2024-03-22T13:34:58.600Z level=INFO source=routes.go:1110 msg="Listening on 127.0.0.1:11434 (version 0.1.29)"
time=2024-03-22T13:34:58.602Z level=INFO source=payload_common.go:112 msg="Extracting dynamic libraries to /tmp/ollama2943820917/runners ..."
t=2024-03-22T13:34:58+0000 lvl=info msg="no configuration paths supplied"
t=2024-03-22T13:34:58+0000 lvl=info msg="using configuration at default config path" path=/root/.config/ngrok/ngrok.yml
t=2024-03-22T13:34:58+0000 lvl=info msg="open config file" path=/root/.config/ngrok/ngrok.yml err=nil
t=2024-03-22T13:34:58+0000 lvl=info msg="starting web service" obj=web addr=127.0.0.1:4040 allow_hosts=[]
t=2024-03-22T13:34:59+0000 lvl=info msg="client session established" obj=tunnels.session
t=2024-03-22T13:34:59+0000 lvl=info msg="tunnel session started" obj=tunnels.session
t=2024-03-22T13:34:59+0000 lvl=info msg="started tunnel" obj=tunnels name=command_line addr=http://localhost:11434 url=https://09f0-34-87-36-160.ngrok-free.app
time=2024-03-22T13:35:04.815Z level=INFO source=payload_common.go:139 msg="Dynamic LLM libraries [cpu cpu_avx cuda_v11 rocm_v60000 cpu_avx2]"
time=2024-03-22T13:35:04.815Z level=INFO source=gpu.go:77 msg="Detecting GPU type"
time=2024-03-22T13:35:04.815Z level=INFO source=gpu.go:191 msg="Searching for GPU management library libnvidia-ml.so"
time=2024-03-22T13:35:04.820Z level=INFO source=gpu.go:237 msg="Discovered GPU libraries: [/usr/lib64-nvidia/libnvidia-ml.so.535.104.05]"
time=2024-03-22T13:35:04.829Z level=INFO source=gpu.go:82 msg="Nvidia GPU detected"
time=2024-03-22T13:35:04.829Z level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-03-22T13:35:04.834Z level=INFO source=gpu.go:119 msg="CUDA Compute Capability detected: 7.5"
t=2024-03-22T13:36:08+0000 lvl=info msg="join connections" obj=join id=3fcefdf0e8ec l=127.0.0.1:11434 r=182.253.54.146:48504
[GIN] 2024/03/22 - 13:36:08 | 403 |      46.987µs |  182.253.54.146 | HEAD     "/"
t=2024-03-22T13:37:27+0000 lvl=info msg="join connections" obj=join id=337841f307f2 l=127.0.0.1:11434 r=182.253.54.146:48464
[GIN] 2024/03/22 - 13:37:27 | 403 |      30.806µs |  182.253.54.146 | HEAD     "/"
t=2024-03-22T13:37:43+0000 lvl=info msg="join connections" obj=join id=18f46d6bb9f4 l=127.0.0.1:11434 r=182.253.54.146:48476
[GIN] 2024/03/22 - 13:37:43 | 403 |      27.165µs |  182.253.54.146 | HEAD     "/"
t=2024-03-22T13:38:53+0000 lvl=info msg="join connections" obj=join id=e4c3ef80532c l=127.0.0.1:11434 r=182.253.54.146:41795
[GIN] 2024/03/22 - 13:38:53 | 403 |      25.822µs |  182.253.54.146 | HEAD     "/"

### This is where i'm stuck because when i run ""C:\Users\bower>$ENV OLLAMA_HOST= "https://09f0-34-87-36-160.ngrok-free.app/"
'$ENV' is not recognized as an internal or external command,
operable program or batch file.

C:\Users\bower>
""