Repository: AUTOMATIC1111/stable-diffusion-webui

Issue:
[Feature Request]: API method to queue a options change
### Is there an existing issue for this?

- [X] I have searched the existing issues and checked the recent builds/commits

### What would your feature do ?

At present, there are two ways to change the model through the API:
1) `POST /sdapi/v1/options` switches the model immediately, which can cause issues for generations in the queue
2) `override_settings` in the generation request will temporarily change the model, but change it back to what it was before

I would like to be able to set the model for upcoming generation requests, but without interfering with the requests already in the queue (ruling out option 1); additionally, I do not want to pay the cost of switching models back to the 'original' model if there's a run of generations with the same model (ruling out option 2).

I'd like a third way that queues up the options change with respect to the existing queue, so that if you have

`generation 1 (SD1.5) | generation 2 (SD1.5)` in the queue, you can issue an options change:
`generation 1 (SD1.5) | generation 2 (SD1.5) | switch to SD2.1 | generation 3 (SD2.1) | generation 4 (SD2.1)`

and generation 1 & 2 remains on SD1.5, while 3 & 4 will use SD2.1 without switching back to 1.5.

The simplest place to make this change would be to add a `queue` boolean parameter to `POST /sdapi/v1/options`, which adds the options switch to the queue instead of actioning it immediately.

### Proposed workflow

1. API user calls `POST /sdapi/v1/options {..., queue: true}`
2. The desired options-change is added to the queue
3. The options are not changed until that position in the queue is reached

### Additional information

I have multiple API clients using the same SD server (a Discord bot, and some experimental programs). I'd like them to be able to switch models and operate entirely independently without communicating with each other. Other API users have solved this by running their own internal queue, so that they can issue `POST /sdapi/v1/options` calls on demand, but I can't do this due to the multiple clients.

Another solution would be to offer a variant of `override_settings` that sets the settings without resetting (`change_settings`?) Either one would be sufficient for my use case (I *always* issue a model change before generating, anyway)

See also #4120 and #5191.


Discussion:
cc @aliencaocao
Unfortunately I do not have enough knowledge on the codes to implement the queue system that you described. As you said, all the API users currently implement the queue themselves. If you have multiple clients, you can also have a centralized middleware server that implements the queue, and only that server sends requests to the API endpoint.

Because a queue system already exists (just without awareness of potential duplicate model switching), if you request for a model change in the middle of a generation, it will actually get queued already and only make the switch after the generation is done. This only works if the generation is also done via the API, since the queue is only for API requests. I do not see why you cannot use the existing behaviour. If you are that conscious about unnecessary model switching, then just using the options endpoint should be your use case.
> Unfortunately I do not have enough knowledge on the codes to implement the queue system that you described.

Ah, apologies; I wasn't suggesting you should implement it, I just wanted to bring it to your attention.

> As you said, all the API users currently implement the queue themselves. If you have multiple clients, you can also have a centralized middleware server that implements the queue, and only that server sends requests to the API endpoint.

Yes, I've considered this as a solution - it's not the cleanest, though, as I'd have to write and run another server to implement a queue on top of an existing queue.

> Because a queue system already exists (just without awareness of potential duplicate model switching), if you request for a model change in the middle of a generation, it will actually get queued already and only make the switch after the generation is done. This only works if the generation is also done via the API, since the queue is only for API requests. I do not see why you cannot use the existing behaviour. If you are that conscious about unnecessary model switching, then just using the options endpoint should be your use case.

The issue I've seen is that this will switch immediately after the current generation, not after all pending generations. With the example in the OP, if you had:

`generation 1 (SD1.5, current) | generation 2 (SD1.5)` 
in the queue, and you issue a options change before issuing two SD2.1 generations, it'll end up doing this:
`generation 1 (SD1.5, current) | switch to SD2.1 | generation 2 (SD2.1) | generation 3 (SD2.1) | generation 4 (SD2.1)` 
instead of:
`generation 1 (SD1.5, current) | generation 2 (SD1.5) | switch to SD2.1 | generation 3 (SD2.1) | generation 4 (SD2.1)`
> switch immediately after the current generation, not after all pending generations

I meant use this behaviour combined with your own queue server so you avoid unnecessary switching
Oh. Yeah, that's possible, but I'd like to avoid that. Might see if I can write a PR to implement this at some point

Resolving patch:
diff --git a/modules/processing.py b/modules/processing.py
--- a/modules/processing.py
+++ b/modules/processing.py
@@ -77,7 +77,7 @@ class StableDiffusionProcessing():
     """
     The first set of paramaters: sd_models -> do_not_reload_embeddings represent the minimum required to create a StableDiffusionProcessing
     """
-    def __init__(self, sd_model=None, outpath_samples=None, outpath_grids=None, prompt: str = "", styles: List[str] = None, seed: int = -1, subseed: int = -1, subseed_strength: float = 0, seed_resize_from_h: int = -1, seed_resize_from_w: int = -1, seed_enable_extras: bool = True, sampler_name: str = None, batch_size: int = 1, n_iter: int = 1, steps: int = 50, cfg_scale: float = 7.0, width: int = 512, height: int = 512, restore_faces: bool = False, tiling: bool = False, do_not_save_samples: bool = False, do_not_save_grid: bool = False, extra_generation_params: Dict[Any, Any] = None, overlay_images: Any = None, negative_prompt: str = None, eta: float = None, do_not_reload_embeddings: bool = False, denoising_strength: float = 0, ddim_discretize: str = None, s_churn: float = 0.0, s_tmax: float = None, s_tmin: float = 0.0, s_noise: float = 1.0, override_settings: Dict[str, Any] = None, sampler_index: int = None):
+    def __init__(self, sd_model=None, outpath_samples=None, outpath_grids=None, prompt: str = "", styles: List[str] = None, seed: int = -1, subseed: int = -1, subseed_strength: float = 0, seed_resize_from_h: int = -1, seed_resize_from_w: int = -1, seed_enable_extras: bool = True, sampler_name: str = None, batch_size: int = 1, n_iter: int = 1, steps: int = 50, cfg_scale: float = 7.0, width: int = 512, height: int = 512, restore_faces: bool = False, tiling: bool = False, do_not_save_samples: bool = False, do_not_save_grid: bool = False, extra_generation_params: Dict[Any, Any] = None, overlay_images: Any = None, negative_prompt: str = None, eta: float = None, do_not_reload_embeddings: bool = False, denoising_strength: float = 0, ddim_discretize: str = None, s_churn: float = 0.0, s_tmax: float = None, s_tmin: float = 0.0, s_noise: float = 1.0, override_settings: Dict[str, Any] = None, override_settings_restore_afterwards: bool = True, sampler_index: int = None):
         if sampler_index is not None:
             print("sampler_index argument for StableDiffusionProcessing does not do anything; use sampler_name", file=sys.stderr)
 
@@ -118,6 +118,7 @@ def __init__(self, sd_model=None, outpath_samples=None, outpath_grids=None, prom
         self.s_tmax = s_tmax or float('inf')  # not representable as a standard ui option
         self.s_noise = s_noise or opts.s_noise
         self.override_settings = {k: v for k, v in (override_settings or {}).items() if k not in shared.restricted_opts}
+        self.override_settings_restore_afterwards = override_settings_restore_afterwards
         self.is_using_inpainting_conditioning = False
 
         if not seed_enable_extras:
@@ -147,11 +148,11 @@ def txt2img_image_conditioning(self, x, width=None, height=None):
 
         # The "masked-image" in this case will just be all zeros since the entire image is masked.
         image_conditioning = torch.zeros(x.shape[0], 3, height, width, device=x.device)
-        image_conditioning = self.sd_model.get_first_stage_encoding(self.sd_model.encode_first_stage(image_conditioning)) 
+        image_conditioning = self.sd_model.get_first_stage_encoding(self.sd_model.encode_first_stage(image_conditioning))
 
         # Add the fake full 1s mask to the first dimension.
         image_conditioning = torch.nn.functional.pad(image_conditioning, (0, 0, 0, 0, 1, 0), value=1.0)
-        image_conditioning = image_conditioning.to(x.dtype)            
+        image_conditioning = image_conditioning.to(x.dtype)
 
         return image_conditioning
 
@@ -199,7 +200,7 @@ def inpainting_image_conditioning(self, source_image, latent_image, image_mask =
             source_image * (1.0 - conditioning_mask),
             getattr(self, "inpainting_mask_weight", shared.opts.inpainting_mask_weight)
         )
-        
+
         # Encode the new masked image using first stage of network.
         conditioning_image = self.sd_model.get_first_stage_encoding(self.sd_model.encode_first_stage(conditioning_image))
 
@@ -463,12 +464,14 @@ def process_images(p: StableDiffusionProcessing) -> Processed:
 
         res = process_images_inner(p)
 
-    finally:  # restore opts to original state
-        for k, v in stored_opts.items():
-            setattr(opts, k, v)
-            if k == 'sd_hypernetwork': shared.reload_hypernetworks()
-            if k == 'sd_model_checkpoint': sd_models.reload_model_weights()
-            if k == 'sd_vae': sd_vae.reload_vae_weights()
+    finally:
+        # restore opts to original state
+        if p.override_settings_restore_afterwards:
+            for k, v in stored_opts.items():
+                setattr(opts, k, v)
+                if k == 'sd_hypernetwork': shared.reload_hypernetworks()
+                if k == 'sd_model_checkpoint': sd_models.reload_model_weights()
+                if k == 'sd_vae': sd_vae.reload_vae_weights()
 
     return res
 
@@ -537,7 +540,7 @@ def infotext(iteration=0, position_in_batch=0):
         for n in range(p.n_iter):
             if state.skipped:
                 state.skipped = False
-            
+
             if state.interrupted:
                 break
 
@@ -612,7 +615,7 @@ def infotext(iteration=0, position_in_batch=0):
                     image.info["parameters"] = text
                 output_images.append(image)
 
-            del x_samples_ddim 
+            del x_samples_ddim
 
             devices.torch_gc()
 
@@ -720,7 +723,7 @@ def save_intermediate(image, index):
 
             samples = torch.nn.functional.interpolate(samples, size=(self.height // opt_f, self.width // opt_f), mode="bilinear")
 
-            # Avoid making the inpainting conditioning unless necessary as 
+            # Avoid making the inpainting conditioning unless necessary as
             # this does need some extra compute to decode / encode the image again.
             if getattr(self, "inpainting_mask_weight", shared.opts.inpainting_mask_weight) < 1.0:
                 image_conditioning = self.img2img_image_conditioning(decode_first_stage(self.sd_model, samples), samples)


Test patch: