Ollama and offline models - Page 1
Forum section: electronics / general

author: paulca
author_rank: Super Contributor
timestamp: August 10, 2025, 10:53:42 am
content: Well this feels a lot more like "home" to me.Most of the OSS models can be downloaded and run locally. Google "Ollama". If you are Windows and dont mind "Community license" there is Myst. That means you can disconnect from the internet and still have full unrestricted MAX token contexts with that model.Further without having to pay for it, you then go to your "Datasheets for Project X" folder, copy and paste them into the bots "RAG" folder, it's library. Now while the model might not understand / contain data on your ICs vLo or iLo you can now ask it, specifying if needed that it should "reference it" and most of them will do exactly that. Using their "learned" context they will navigate the datasheet find the relevant excerpts etc. It will then provide you will a reference, excerpt and link (if using the OpenUI kit).Now the downside. Hardware. These things eat hardware. Several metrics are important. CPU or GPU core count/speed and VRAM/RAM size and speed."Deepseek R1" for example comes in different sizes, these are "quantized" from large models.Anywhere from 1 billion parameters which will run on a Raspberry PI all the way up to the 405B parameters which Dave Plumber on YT demonstrated will bring a maxed out £50k Dell workstation to it's knees in tears.I tried Llamma (Meta) and gemma(Google) on the gaming rig with a 3080/10Gb card and with 12B sized models it was faster than chat gpt online, although a far bit dumber. Fans kicked off fully like when running a gaming benchmark or bit coin mining.The problem is, to go higher causes issues as in most circumstances, presently, the model has to fit into either GPU VRAM and be "GPU accelerated" or run in CPU RAM and be CPU slow. That machine has 64Gb of RAM, so I could run 60B+ models but with the CPU and much, much slower.I then tried deepseek R1 @12B on linux in a VM. I asked it a small prompt:"Can you help me archive my email off GMail to local files?"It's been going at it for an hour now. What I see now is what is behind the "Thinking" on the web UI just in the terminal. It's fascinating to watch. It has produced a set of steps for a python IMAP client with security certs and OAUTH, then corrected itself that it wouldn't work because of some problem with the Google OAuth so it went back and tried it again. About 3 or 4 attempts later it seemed happy and carried on with the actual python code.The creepy thing is.... it's referring to "itself" by the collective 3rd person a lot."I think the user wanted x, butoursolution might not cover what they asked,weshould try the other approach...."I just watched it go through a load of Google API python setup, then counter itself and revert to focusing on IMAP again.Its writing a novel! At some point I believe it will settle and converge on its answer, resummarise it as it's final answer. I just dont know how long that will take on CPU in 16Gb VM.QuoteQuoteIt's not just writing bad code and handing it too you, it's writing it, realising its wrong and reiterating.It seems sort of stuck because it hasn't been able to produce what it thinks is a "Working google API or IMAP login" yet. So everytime it tries to write code to access the API it keeps harassing itself that it can't access it yet so it's not going to work. Then it tries the security again, failed to find a solution and goes back to writing IMAP or GAPI code again.Its presently having a deep argument with itself and it's inability to provide a solution.It's getting surreal:Quote