Fix bias logic to enable QLoRA finetuning

by winglian - opened Mar 28

base: refs/heads/main

←

from: refs/pr/5

Discussion Files changed

+10

-4

winglian

Mar 28

when using the qlora technique, dt_proj doesn't have a bias attribute, resulting in an AttributeError. This change allows for qlora finetuning with an approximate train loss ~1-ish.

Fix bias logic to enable QLoRA finetuning871ab2b1

tomeras1

AI21 org Mar 28

Nice! Thank you :)
A few comments:

Can you add a comment in code explaining why this change is needed?
Don't you also need to edit line 953? IIUC, in case of qlora there is no bias attribute so time_proj_bias in line 953 will be None, which is not what we want..

tomeras1

AI21 org Mar 28

Closed because fixed in a different PR

tomeras1 changed pull request status to closed Mar 28

ArthurZ

Mar 28

https://github.com/huggingface/peft/pull/1530 Should have fixed these in latest peft

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment