Repository: AUTOMATIC1111/stable-diffusion-webui

Issue:
[Bug]: New SHA256 hash takes extremely long time up to a point of of model load being unusable
### Is there an existing issue for this?

- [X] I have searched the existing issues and checked the recent builds/commits

### What happened?

Newly added sha-256 hash takes extremely long time to calculate on model load up to a point where loading appears to hang (i've restarted server twice before i even let it run until completion)  

Previously switching to a new model was sub 10 sec, now switching to a new model (that does not have hash stored already) takes 100-150 sec (and this is a high end system)!

And to make it worse, messages about hash calculation are only printed **after** it has been calculated, there is no progress info or anything to indicate system is actually doing anything for 2 min!


### Steps to reproduce the problem

1. Switch to a new model and wait for completion - it takes forever


### What should have happened?

Model load should **never** take over 2 minutes to complete.

### Commit where the problem happens

f8c512478568293155539f616dce26c5e4495055

### What platforms do you use to access UI ?

Windows, Linux

### What browsers do you use to access the UI ?

Google Chrome, Microsoft Edge

### Command Line Arguments

```Shell
--api --xformers
```


### Additional information, context and logs

Console log showing model load taking 142 seconds!

```text
Calculating sha256 for /home/vlado/dev/automatic/models/Stable-diffusion/mood-beautyreal-v01.ckpt: bcc0afd3b264ea028928187f56f70840f8d87ccf283b020982beba35d9c7e4ef
Loading weights [bcc0afd3b2] from /home/vlado/dev/automatic/models/Stable-diffusion/mood-beautyreal-v01.ckpt
Couldn't find VAE named vae-ft-mse-840000-ema-pruned; using None instead
Applying xformers cross attention optimization.
Weights loaded in 142.6s.
```



Discussion:
Odd, how big is that model file? I'm getting nowhere near the time you do. On a 8GB file:
```
Calculating sha256 for C:\tools\Ai\stable-diffusion-webui\models\Stable-diffusion\protogenX58RebuiltSc_10.ckpt: 710fc74d4cf3245cf5f1664f25502da0c7750f80e8bbbb56929e710043d84efa
Loading weights [710fc74d4c] from C:\tools\Ai\stable-diffusion-webui\models\Stable-diffusion\protogenX58RebuiltSc_10.ckpt
Couldn't find VAE named vae-ft-mse-840000-ema-pruned.vae; using None instead
Applying xformers cross attention optimization.
Model loaded in 16.2s (0.5s create model, 15.6s load weights).
```

Is your model on ssd?
I'm getting 80sec for 2gb model, 100sec for 4gb model and 140sec for 7gb model.

Disk is as fast as possible, over 1. 5GBytes/sec.

Only difference is that disk is mounted over network loopback (shared between VMs), so maaaaaaybe sha method does some strange byterange access so it does full access on each range? But if it does, that's a broken file read inside sha method.

And just to confirm - I was having sub 10sec times before the change - thats 10x slowdown!

It is taking longer for me too.. I also have the models on my ssd.. it's taking like 5 more times to load now...

Also, the hash number has disappeared from the model's name list? 
> Only difference is that disk is mounted over network loopback (shared between VMs), so maaaaaaybe sha method does some strange byterange access so it does full access on each range?

Can you try changing this function in hashes.py from:
```
def calculate_sha256(filename):
    hash_sha256 = hashlib.sha256()

    with open(filename, "rb") as f:
        for chunk in iter(lambda: f.read(4096), b""):
            hash_sha256.update(chunk)

    return hash_sha256.hexdigest()

```
to:
```
def calculate_sha256(filename):
    import mmap

    hash_sha256 = hashlib.sha256()

    with open(filename, "rb") as f:
        mmapped = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ)
        while True:
            chunk = mmapped.read(4096)
            if not chunk:
                break
            hash_sha256.update(chunk)

    return hash_sha256.hexdigest()

```
Perhaps memory-mapping can help in this case.
I have some Error - "Couldn't find VAE named vae-ft-mse-840000-ema-pruned; using None instead"
what is the problem?
I figured it out, the next git pull in the settings reset the link to VAE simply
@AlUlkesh 

i've tried your code with `mmap`, no change in performance.
looking at user vs sys time and actual io performance, 99% of time is spent inside python and my actual disk i/o is really slow at only 50mb/sec.

then i've switched to `byteranges` and omg its sooooo much faster - down from 100sec to 4sec!  
(and my disk i/o went from 50mb/sec to 600mb/sec)

```python
    import io
    chunk = bytearray(1024 * 1024)
    with io.open(filename, "rb", 0) as f:
        while True:
            numread = f.readinto(chunk)
            if not numread:
                break
            hash_sha256.update(chunk)
```

i'll create a pr for this.
I did a fresh install and now loading models speed is normal again... but the hash is still gone
@vladmandic Great, for me this cuts down the hash calculation from 9 to 4 seconds on a 8GB model.

But I seem to get a different hash then before (with model protogenX58RebuiltSc_10.ckpt)

Before I got 710fc74d4cf3245cf5f1664f25502da0c7750f80e8bbbb56929e710043d84efa
Now it's 4d69515e3de52a4b0259ee5922c833f5e13deb8efb6c9ad500f1dda173c892d0

Checking with an external tool, 710fc74d4cf3245cf5f1664f25502da0c7750f80e8bbbb56929e710043d84efa seems to be correct.

Any idea what could be responsible for the change?
@vladmandic Oh, I think I know what it is. The new version always uses the full 1024 * 1024 bytes to update the sha256. That means the last chunk will pass bytes that have not been read.

Something like this should solve the issue:
`hash_sha256.update(chunk[:numread])`

- [ ] @AlUlkesh 

i've started experimenting with 

- `open` + `read`
- `open` + `mmap.read`
- `io.open` + `read`
- `io.open` + `readinto`

and different blocksizes.

the fastest combination is simplest routine as-in, but with larger blocksize.
basically, with 4k chunks python is taking forerver in inner loop and actual io is not even close to saturate disk
and with 1mb chunk, performance goes up 10x (my actual hash calc is down to ~3sec)

i don't see why we would use 4kb chunks to start with? lets just do read with larger blksize and thats it.


Resolving patch:
diff --git a/modules/hashes.py b/modules/hashes.py
--- a/modules/hashes.py
+++ b/modules/hashes.py
@@ -34,9 +34,10 @@ def cache(subsection):
 
 def calculate_sha256(filename):
     hash_sha256 = hashlib.sha256()
+    blksize = 1024 * 1024
 
     with open(filename, "rb") as f:
-        for chunk in iter(lambda: f.read(4096), b""):
+        for chunk in iter(lambda: f.read(blksize), b""):
             hash_sha256.update(chunk)
 
     return hash_sha256.hexdigest()


Test patch: