Repository: ollama/ollama
Pull request: server/sched.go:  Added support for grouping GPUs ( significant VRAM reduction, Power-Consumption + Low-PCIe-speed performance gains)

This patch modifies Ollama to allow grouping GPUs to memory-fit to the requested model, instead of the former algorithm of using one GPU <OR> distributing over all available GPUs. 

### Benefits:
* Lower amount of (PCIe-)bus communication between GPUs - especially when they are not very high speed  
* Allowing unallocated GPUs to get into power-saving mode. 
* Significantly reduce VRAM allocation when using more than 2 GPUs in a system (see below) 
* Due to the reduced memory allocation, you can run more models simultaneously.  

### How to use: 
* OLLAMA_SCHED_SPREAD=0 (or keep it unset) to use only one GPU, and then evenly distribute the load across all GPUs. (old default behavior) 
* OLLAMA_SCHED_SPREAD=1 (or set to anything else) to evenly distribute the load across all GPUs. (old default behavior)
* OLLAMA_SCHED_SPREAD=2 to keep the number of GPUs to a minimum and only use as many as needed to run the model. 


### Tests and Environment: 
Like so many in the home-lab community, my Ollama server is built on a budget, which resulted in a low-power optimized server with 7 GPUs (Nvidia RTX 3060 - 12GB) with a total of 72GB VRAM. 

When using models which require around 30-40 GB, all GPUs have to run, which results in a higher power consumption and a huge overhead in VRAM allocation. With this patch, just a few models are in use, which powers down all other GPUs to around 3 watts. 
Since the server (Ryzen 5500G / Asus Prime B450)  is not enterprise grade, the PCIe bus far slower, which also slows down the utilization when using too many GPUs with one single model. 

### Results of Memory-Allocation Overhead when spreading a model over more than 2 GPUs: 
In this example, we configure Ollama to num_parallel=2, num_ctx=8192  

Model: gemma3:4b -  a2af6cc3eb7f   - 3.3 GB 
Overall required VRAM when spreading over:
1x GPU: 5.7 GiB 
2x GPU: 7.8 GiB
3x GPU: 9.3 GiB
4x GPU: 10.8 GiB
5x GPU: 12.3 GiB
6x GPU: 13.9 GiB
7x GPU: 15.4 GiB

a more realistic situation could be qwen3:30b - 0b28110b7a33   - 18 GB , which exceeds the home-lab gpus. 
3x GPU: 27.2 GiB
4x GPU: 30.1 GiB
5x GPU: 32.9 GiB
6x GPU: 35.8 GiB
7x GPU: 38.7 GiB 


### Results of Power consumption measured "at the wall" and eval rates 

Qwen3:14b - bdbd181c33f2 - OLLAMA_SCHED_SPREAD unset:  450 Watt with 27 fps  
Qwen3:14b - bdbd181c33f2 - OLLAMA_SCHED_SPREAD=1:  450 Watt with 27 fps  
Qwen3:14b - bdbd181c33f2 - OLLAMA_SCHED_SPREAD=2:  **310 Watt with 81 fps**  

Qwen3:30b - 0b28110b7a33 - OLLAMA_SCHED_SPREAD unset:  393 Watt with 44 fps  
Qwen3:30b - 0b28110b7a33 - OLLAMA_SCHED_SPREAD=1:  393 Watt with 44 fps  
Qwen3:30b - 0b28110b7a33 - OLLAMA_SCHED_SPREAD=2:  **320 Watt with 51 fps**  

(Yes, Qwen3:14b is far more inefficient on my system than the slightly younger 30b variant) 

gemma3:4b - a2af6cc3eb7f - OLLAMA_SCHED_SPREAD unset: 250 Watt with 70 fps 
gemma3:4b - a2af6cc3eb7f  - OLLAMA_SCHED_SPREAD=1 : 389 Watt with 55 fps 
gemma3:4b - a2af6cc3eb7f  - OLLAMA_SCHED_SPREAD=2 : **250 Watt with 70 fps** 

(gemma3:4b with num_ctx of 8192 also resulted in just utilizing 1 GPUs with OLLAMA_SCHED_SPREAD=2 similar to the unset OLLAMA_SCHED_SPREAD as one GPU is enough to fit the whole model)




Changed files:
filename: server/sched.go
patch: @@ -758,8 +758,6 @@ func (a ByDurationAndName) Less(i, j int) bool {
 // If numParallel is <= 0, this will attempt try to optimize parallelism based on available VRAM, and adjust
 // opts.NumCtx accordingly
 func pickBestFullFitByLibrary(req *LlmRequest, f *ggml.GGML, gpus discover.GpuInfoList, numParallel *int) discover.GpuInfoList {
-	var estimatedVRAM uint64
-
 	var numParallelToTry []int
 	if *numParallel <= 0 {
 		// If no specific parallel setting was provided, try larger then smaller, always end with 1
@@ -769,39 +767,51 @@ func pickBestFullFitByLibrary(req *LlmRequest, f *ggml.GGML, gpus discover.GpuIn
 	}
 
 	for _, gl := range gpus.ByLibrary() {
-		var ok bool
 		sgl := append(make(discover.GpuInfoList, 0, len(gl)), gl...)
 
 		// TODO - potentially sort by performance capability, existing models loaded, etc.
 		// TODO - Eliminate any GPUs that already have envconfig.MaxRunners loaded on them
-		// Note: at present, this will favor more VRAM over faster GPU speed in mixed setups
+		// Note: at present, this will favor most current available VRAM descending and ignoring faster GPU speed in mixed setups
 		sort.Sort(sort.Reverse(discover.ByFreeMemory(sgl)))
 
-		// First attempt to fit the model into a single GPU
-		for _, p := range numParallelToTry {
-			req.opts.NumCtx = req.origNumCtx * p
-			if !envconfig.SchedSpread() {
-				for _, g := range sgl {
-					if ok, estimatedVRAM = llm.PredictServerFit([]discover.GpuInfo{g}, f, req.model.AdapterPaths, req.model.ProjectorPaths, req.opts, p); ok {
-						slog.Info("new model will fit in available VRAM in single GPU, loading", "model", req.model.ModelPath, "gpu", g.ID, "parallel", p, "available", g.FreeMemory, "required", format.HumanBytes2(estimatedVRAM))
+		if !envconfig.SchedSpread() {
+			for _, p := range numParallelToTry {
+				req.opts.NumCtx = req.origNumCtx * p
+				// Try to pack into as few GPUs as possible, starting from 1 GPU
+				for numGPUs := 1; numGPUs <= len(sgl); numGPUs++ {
+					gpuSubset := sgl[:numGPUs]
+					ok, estimatedVRAM := llm.PredictServerFit(gpuSubset, f, req.model.AdapterPaths, req.model.ProjectorPaths, req.opts, p)
+
+					if ok {
+						slog.Info("new model will fit in available VRAM across minimum required GPUs, loading",
+							"model", req.model.ModelPath,
+							"library", sgl[0].Library,
+							"parallel", p,
+							"required", format.HumanBytes2(estimatedVRAM),
+							"gpus", numGPUs)
 						*numParallel = p
-						return []discover.GpuInfo{g}
+						return gpuSubset
 					}
 				}
 			}
-		}
-
-		// TODO future refinements
-		// - if multiple Libraries, see if any single GPU in any Library will fit
-		// - try subsets of GPUs instead of just falling back to 1 or all in a family
-
-		// Now try all the GPUs
-		for _, p := range numParallelToTry {
-			req.opts.NumCtx = req.origNumCtx * p
-			if ok, estimatedVRAM = llm.PredictServerFit(sgl, f, req.model.AdapterPaths, req.model.ProjectorPaths, req.opts, p); ok {
-				slog.Info("new model will fit in available VRAM, loading", "model", req.model.ModelPath, "library", sgl[0].Library, "parallel", p, "required", format.HumanBytes2(estimatedVRAM))
-				*numParallel = p
-				return sgl
+		} else {
+			// TODO future refinements
+			// - if multiple Libraries, see if any single GPU in any Library will fit
+			// - try subsets of GPUs instead of just falling back to 1 or all in a family
+
+			// Now try all the GPUS (OLLAMA_SCHED_SPREAD is set)
+			for _, p := range numParallelToTry {
+				req.opts.NumCtx = req.origNumCtx * p
+				if ok, estimatedVRAM := llm.PredictServerFit(sgl, f, req.model.AdapterPaths, req.model.ProjectorPaths, req.opts, p); ok {
+					slog.Info("new model will fit in available VRAM, loading",
+						"model", req.model.ModelPath,
+						"library", sgl[0].Library,
+						"parallel", p,
+						"required", format.HumanBytes2(estimatedVRAM),
+						"gpus", len(sgl))
+					*numParallel = p
+					return sgl
+				}
 			}
 		}
 	}