FantasiaFoundry

/

GGUF-Quantization-Script

like 40

Might just add an option, keeping HF files was not part of the initial use case.

It's up to you of course, in my opinion it should not be deleting user files at all, unless the script itself downloaded those files, which it did not. So in a big sense this is over-stepping quite a lot.

In my experience so far with merging and quantizing, I always start with just one GGUF file like q4_km to test if the merged model even works, as it may be too big to run as HF merged model and there is no way to test it before that, and then it would just delete the HF model after doing one quant anyhow.

The use case with wiping the HF model seems pretty limited and narrow, it's more like an edge case for someone who is absolutely confident they do not want this HF model on their system anymore. It seems more reasonable that the user would decide what to do with a complete HF model they have, not the script, since again, the script did not download the model, the user did and manually placed it in the models folder. But really, it doesn't bother me as I have just deleted that from the script, but it can catch other users off-guard, and it caught me off-guard when I updated it :P

Besides that, it's a fantastic script and this is pretty much the one thing I changed in it. If you make it an option, imho it should be turned off by default, with some note in the readme that it can be switched on if you want to auto-delete models. Plus, in this case it's very easy to track where the model is (as you placed it manually in the folder) and it's quite easy for the user to manually delete it with 2 clicks, it's not hidden in a temp folder like mergekit likes to do.

in my opinion it should not be deleting user files at all, unless the script itself downloaded those files, which it did not. So in a big sense this is over-stepping quite a lot

The script uses huggingface_hub to download the HF models. I didn't expect deleting the local HF model folder after getting the F16 GGUF was a big deal as the F16 GGUF is really the only thing you need to make quants, and the model remains cached in the .cache folder so at least the download step is skipped, but I can see the points of improvement, since this mostly just started as a personal thing for me it wasn't really taken into account for other uses/setups. Merging and tuning wasn't in the initial plan/use case for this.

the script did not download the model

But yea, the script will download the models. That's the first step.

Get model > Convert to GGUF F16 > Delete HF > Get Imatrix > Make Quants.

the user did and manually placed it in the models folder

Now that was NOT an expected way to do this for me xD

in this case it's very easy to track where the model is (as you placed it manually in the folder)

The script does download HF models into the models folder and they are deleted after the F16 GGUF is done.

The script will not download the model if it exists in the root folder, which is what I have been doing since I:

1. create a new merge and
2. place the merged model in the /models folder of the GGUF script

The download is totally optional, although you are totally right that it does that if you don't have the model locally. The point is that's not necessary at all, and will not be the use case at all if you created a new merge on your computer. (It makes no sense to first upload the merge, and then re-download the whole thing to your computer when doing the GGUF quantization)

So really, it only makes sense if you actually want to do quants of 3rd party models that you did not create, and the use case changes quite drastically if you do the merges locally, and do want to keep the HF model. Deleting the HF folder is a big deal if this is your only copy of the merge you did, which you then would like to perhaps upload manually as a HF model after the quants are ready (that's how I use it anyhow). Technically not the hugest deal, of course, as the actual merge won't take long to re-do, but it may for some computers.

I guess the way we've been using it is different and that's why I had this feedback, otherwise your use case makes sense for 3rd party models as the HF is indeed likely not going to be needed for those users as they just want the quants.

However, I am not sure if this deleting is totally safe from errors, if there was a problem with the quantization it would still end up deleting the HF and then it would mean re-download if the user needs to re-try it.

if there was a problem with the quantization it would still end up deleting the HF and then it would mean re-download if the user needs to re-try it

Not really, the downloaded HF model is cached in the .cache/huggingface/hub as discussed, so in case of a retry, these files will be symlinked from the cache instead of actually re-downloaded (of course you'll have a bit of IO overhead in this process but it's not ganna be as long as the real download).

This might be a bit of a scope of usage mismatch as this was really only meant to help people convert third party models to GGUF locally if there's not already one on HF done for them, the time frame was a bit after TheBloke's absense was showing it would continue for longer. I started doing my own quants for personal use and sharing them. Since I wanted Imatrix quants that's how it happened.

Locally created merges will not be in the HF hub cache. But if the use case is entirely dedicated to converting hub models that you always download, then I understand. At the end of the day it's a non-issue technically because anyone could just delete those lines.

With merges I also considered that most people doing those would perform them on the cloud and then upload to HF actually? As in Google Colab or another dedicated cloud hardware provider, so the process of downloading would be the same. I may have just assumed too much there.

Real model gamers do everything locally 😎I must've went through 10TB of data already, each upload for me is between 50 and ~100GB depending on the model size and whether Q8 is also included (alongside the new HF model itself). I didn't build this PC for nothing 😂When I upgrade the GPU it's gonna be even crazier, probably, I might need another 2TB stick as well...

I can respect that.
I'm not a model gamer, I'm just a Lewd regular gamer.

All my merges are done locally to, but I end up downloading models, deleting them and re-downloading the old models for a new merge. My network usage went up 4tb from all the reinstalling models. I just don't trust leaving my hf cache on a HDD with 40k+ hours

I'm actually an ultra noob model gamer but trying to sound cool is important :D

Since my connection is unmetered, I never thought about the amount traffic but we do use a lot if we thing about it, haha.

I'd like to know how much bandwidth/storage thebloke uses, it must be nearing or in 100's of tbs

Imagine having a metered connection in 2024 😭😭😭

I am planning on getting 1gigabit when I move back into my old place in a few months :`)

We did get lucky with speeds here, every connection in the country got a 3x speed increase for free. Yippee!
But I'm moving cities to a smaller one and their 4gigabit lines cost the same as a gig line here because the current address is commercial, so beeg fees. A fibre phone line costs $30 here

Imagine having a metered connection in 2024 😭😭😭

I hear this complaint a lot from US folks in monopolized areas. ISPs are even outright blocking websites they deem "dangerous"... 💀

Talk about anti consumer.

In the US some of the major ISPs block catbox, and won't give a reason why.
Even here I run into blocked websites when not using a VPN, so frequently I just use one permanently because it gets so annoying
Edit - forgot an "s" :')

I'm lucky to exist in the EU, the internet has been pretty sick and cheap here for ~20 years now. I pay right about 15$/month for half gigabit, and it's not on contract (I can end it at any time). On contract, it's even cheaper! But I am renting atm, so I went like that.

Wow! It's $138 NZD/$83 USD here for 300/300 on contract 😭
Everything here is crazy expensive though so it fits

@ABX-AI As a South American I can relate. Pretty much the same pricing. 20$/month for 600/600.

🚀

@saishf When they started blocking Nyaa.si there so knew things were bad. People having to use VPNs to not get in trouble with Torrenting for me is absolutely insane.
That's a dystopia.

But that's because of the lack of competition (and of course dubious priorities from lawmakers in bed with corporations), I can get 3 other options with the same price range here if my ISP acts funny. The bargaining power is with the consumer. The US is often a duopoly that has no intention of actually competing when it comes to ISPs.

@saishf When they started blocking Nyaa.si there so knew things were bad. People having to use VPNs to not get in trouble with Torrenting for me is absolutely insane.
That's a dystopia.

But that's because of the lack of competition (and of course dubious priorities from lawmakers in bed with corporations), I can get 3 other options with the same price range here if my ISP acts funny. The bargaining power is with the consumer. The US is often a duopoly that has no intention of actually competing when it comes to ISPs.

The most stupid thing is, every copyright notice they send costs them $25 USD. The movie is only $15 USD, what is the point?
It'd also help if everything wasn't so expensive that people can't afford to pay for nice things like movies constantly.

Alright! So you'll be prompted now about deleting or not the original model and it defaults to 'no' to prevent accidents when mashing Enter.