How do i create similar small shards for other models I want to offload to GPU?
#1
by
SachinKaushik
- opened
Hi Abhishek, thanks for sharing this. Sorted the issue of having just 13Gigs of RAM Available on my system. I was wondering about the process of creating shards. If there is any resource you can point me to, it would be awesome.
model = AutoModelForCausalLM.from_pretrained(....)
tokenizer = AutoTokenizer.from_pretrained(...)
model.save_pretrained(path, max_shard_size="3GB")
tokenizer.save_pretrained(path)
and then you can push to hub
Thanks