North-ML1

/

Aurora-Proelia

like 0

Follow

North ML 12

Ill try

See benchmarks

Can you tell me in %, acc norm of Hellaswag,PIQA and Arc easy

check readme

You didn't give arc easy or poqa

Only hellaswag

oh

does it beat it

its fine ill benchmark

I CANNOT benchmark IT you Did not make a implementation

wdym

to test check readme and use question/answer format

do you want a standard chat template?

Then show me how to run a prompt trough it

k I made a chatml compatible repo

Im not asking about chat template, there is no inference code

oh I fixing that

fixed

Yeah but I now cant benchmark, either you run piqa, arc easy, base bench or not

idk

But on hellaswag alone, its 7% worse

benchmark now

idk what happened but so when i benchmarked it

ARC Easy
30.64
acc_norm,none
0.306397
PIQA
55.77
acc_norm,none
0.557671
ARC Challenge
26.11
acc_norm,none
0.261092
HellaSwag
27.64
acc_norm,none
0.276439

By that, its worse than my 3M model?

@arthu1

how many training tokens?

28 training tokens/parameter and a bunch of sft

bruh

28 tokens per parameter?

That explains ARC Easy
30.64
acc_norm,none
0.306397
PIQA
55.77
acc_norm,none
0.557671
ARC Challenge
26.11
acc_norm,none
0.261092
HellaSwag
27.64
acc_norm,none
0.276439

@Banaxi-Tech It's not my fault I don't have so much compute. :(

@arthu1 ik, just wanted to tell you if you dont know why in the future

ty where should I go for compute tho

Idk, Google cloud has free compute grants that give like 16x tpu clusters but they're currently closed but you could go on the waitlist

@Banaxi-Tech CPT'ing on a separate checkpoint pre-sft and not overfitting this time!