Repository: huggingface/diffusers

Issue:
LCM Scheduler isn't perfect in uniformly spreading the timesteps across the full range
### Describe the bug

Timesteps for nSteps=26 using the current scheduler
> [999, 979, 959, 939, 919, 899, 879, 859, 839, 819, 799, 779, 759,
    739, 719, 699, 679, 659, 639, 619, 599, 579, 559, 539, 519, 499]

Notice that it comes up short of coming near zero.
My modification is still linear but across the full range and still keeping the individual timesteps as multiples of 20 away from 999:
> [999, 959, 919, 879, 839, 799, 759, 739, 699, 659, 619, 579, 539,
    499, 459, 419, 379, 339, 299, 259, 239, 199, 159, 119,  79,  39]

Note that at 25 steps and at a few other step values both schedules result in the same timesteps and same image.  In the "Asian woman with fancy bikini" image below with the old scheduler it loses the sky, clouds and sand/sea edge when going from 25 steps to 26 steps.  Perhaps due to the lopsided schedule???

![Screenshot from 2023-11-15 21-23-18](https://github.com/huggingface/diffusers/assets/116415616/6401bd21-812f-4845-b8d3-f1a5c0d638ab)


### Reproduction

For a hack to test my modified scheduler I took:
> diffusers/schedulers/scheduling_lcm.py:set_timesteps()
and after the old scheduler created the self.timesteps tensor I just overrode it with:
```
    nSteps = len(self.timesteps)
    delta = 999. / nSteps
    for i in range(nSteps):
        self.timesteps[i] = 999 - 20 * int((i * delta + 10) / 20)
```
Obviously this is hard coded for the 20 increment delta requirement Simian told me of when I reported this to him.


### Logs

_No response_

### System Info

This is the diffusers version with LCM now integrated fully.

### Who can help?

_No response_


Discussion:
The origin LCM scheduling is indeed flawed.
Should we open a PR to fix it? :-) 
Sure, we can. @aifartist how about you open a PR? It's your discovery.

Resolving patch:
diff --git a/src/diffusers/schedulers/scheduling_lcm.py b/src/diffusers/schedulers/scheduling_lcm.py
--- a/src/diffusers/schedulers/scheduling_lcm.py
+++ b/src/diffusers/schedulers/scheduling_lcm.py
@@ -373,10 +373,11 @@ def set_timesteps(
             )
 
         # LCM Timesteps Setting
-        # Currently, only linear spacing is supported.
-        c = self.config.num_train_timesteps // original_steps
-        # LCM Training Steps Schedule
-        lcm_origin_timesteps = np.asarray(list(range(1, int(original_steps * strength) + 1))) * c - 1
+        # The skipping step parameter k from the paper.
+        k = self.config.num_train_timesteps // original_steps
+        # LCM Training/Distillation Steps Schedule
+        # Currently, only a linearly-spaced schedule is supported (same as in the LCM distillation scripts).
+        lcm_origin_timesteps = np.asarray(list(range(1, int(original_steps * strength) + 1))) * k - 1
         skipping_step = len(lcm_origin_timesteps) // num_inference_steps
 
         if skipping_step < 1:
@@ -385,9 +386,13 @@ def set_timesteps(
             )
 
         # LCM Inference Steps Schedule
-        timesteps = lcm_origin_timesteps[::-skipping_step][:num_inference_steps]
+        lcm_origin_timesteps = lcm_origin_timesteps[::-1].copy()
+        # Select (approximately) evenly spaced indices from lcm_origin_timesteps.
+        inference_indices = np.linspace(0, len(lcm_origin_timesteps), num=num_inference_steps, endpoint=False)
+        inference_indices = np.floor(inference_indices).astype(np.int64)
+        timesteps = lcm_origin_timesteps[inference_indices]
 
-        self.timesteps = torch.from_numpy(timesteps.copy()).to(device=device, dtype=torch.long)
+        self.timesteps = torch.from_numpy(timesteps).to(device=device, dtype=torch.long)
 
         self._step_index = None
 


Test patch:
diff --git a/tests/schedulers/test_scheduler_lcm.py b/tests/schedulers/test_scheduler_lcm.py
--- a/tests/schedulers/test_scheduler_lcm.py
+++ b/tests/schedulers/test_scheduler_lcm.py
@@ -84,7 +84,7 @@ def test_time_indices(self):
 
     def test_inference_steps(self):
         # Hardcoded for now
-        for t, num_inference_steps in zip([99, 39, 19], [10, 25, 50]):
+        for t, num_inference_steps in zip([99, 39, 39, 19], [10, 25, 26, 50]):
             self.check_over_forward(time_step=t, num_inference_steps=num_inference_steps)
 
     # Override test_add_noise_device because the hardcoded num_inference_steps of 100 doesn't work