Ollama

Related articles
General-purpose computing on graphics processing units
llama.cpp
Vulkan
Ollama
is an application which lets you run offline large language models locally.
Installation
Install
the
ollama
package, which provides a daemon, command line tool, and CPU inference.
For GPU inference:
Install
ollama-cuda
for inference with
CUDA
.
Install
ollama-rocm
for inference with
ROCm
.
Install
ollama-vulkan
for inference with
Vulkan
(
experimental
).
Next,
enable/start
ollama.service
. Then, verify Ollama's status:
$ ollama --version
If it says
Warning: could not connect to a running Ollama instance
, then the Ollama service has not been run; otherwise, the Ollama service is running and is ready to accept user requests.
Next, verify that you can run models. The following command downloads the latest
270M parameter model of Gemma 3
and returns an Ollama prompt that allows you to talk to the model:
$ ollama run gemma3:270m
>>> Send a message (/? for help)
Usage
The Ollama executable does not provide a search interface. There is no such command as
ollama search
. To search for a model, you need to visit their
search page
.
To run a model:
$ ollama run
model
To stop a model:
$ ollama stop
model
To update a model:
$ ollama pull
model
To remove a model:
$ ollama rm
model
To view locally available models:
$ ollama list
Troubleshooting
ROCm is not utilizing my AMD GPU
You may have used utilities like
amdgpu_top
to monitor the utilization of your GPU during an Ollama session, but only to notice that your GPU has not been used at all.
Without configuration,
ROCm
simply ignores unsupported GPUs, causing everything to be computed on CPU.
Note
Verify supported GPUs by consulting
ROCm System Requirements
.
To work this around, create a
drop-in file
for
ollama.service
:
/etc/systemd/system/ollama.service.d/override_gfx_version.conf
[Service]
Environment="HSA_OVERRIDE_GFX_VERSION=X.Y.Z"
Where
X.Y.Z
is dependent to the GFX version that is shipped with your system.
To determine which GFX version to use, first make sure
rocminfo
has already been installed. It should be pulled in to your system as a dependency of
rocblas
, which is itself a dependency of
ollama-rocm
.
Next, query the actual GFX version of your system:
$ /opt/rocm/bin/rocminfo | grep amdhsa
You need to remember the digits printed after the word
gfx
, because this is the actual GFX version of your system. The digits are interpreted as follows:
If the digits are 4-digit, they are interpreted as
XX.Y.Z
, where the first two digits are interpreted as the
X
part.
If the digits are 3-digit, they are interpreted as
X.Y.Z
.
Then, find all installed
rocblas
kernels:
$ find /opt/rocm/lib/rocblas/library -name 'Kernels.so-*'
You need to set
X.Y.Z
to one of the available versions listed there. The rules are summarized as follows:
For the
X
part, it must be strictly equal to the actual version.
For the
Y
part, mismatch is allowed, but it must be no greater than the actual version.
For the
Z
part, mismatch is allowed, but it must be no greater than the actual version.
After setting the correct
X.Y.Z
, perform a
daemon-reload
and
restart
ollama.service
.
Then, run your model as usual. You may wish to monitor GPU utilization with
amdgpu_top
again.
Models are not removed after uninstalling Ollama
You can manually remove the model files. They are stored in
/var/lib/ollama/blobs
.
See also
Ollama Blog
Ollama Docs