None of the UL2 based models have been evaluated.

#114
by felixz - opened

I like to see Flan-T5 compared to the rest.

Open LLM Leaderboard org

Hi @felixz , are they in the request queue?

@clefourrier
So I don't see google/flan-ul2 in the request queue but when i tried to submit it I got error saying "Model has already been submitted".
I guess flan-ul2 is the model from Google wich is best model claimed in their paper.
An older model from last year google/flan-t5-xxl would be nice to see and it has been in queue for a while.

Open LLM Leaderboard org

Thank you for reporting, investigating today

Open LLM Leaderboard org

@felixz After further investigation, it would seem we are not supporting them at the moment.
We will do our best to add support fast :)

clefourrier changed discussion status to closed

I guess I understand why.. without some more public research and continued innovation by Google there is no momentum to revive the T5 architecture. The initial research did point to T5 outperforming GPT on instructional data. Still Google closed up all their research since and we don't exactly know where that went. PaLM2 could be T5 architecture but those details were never reviled by Google.

Sign up or log in to comment