open llama sgpt?
Is it possible to use your procedure to make sentence embeddings from the new open source llms?
SGPT and sentence-t5 were long sota embeddings.
Take one of the better, new llms.
With 4 bit quantization, can one create like sota mpt-30b sentence embeddings that run on consumer hardware?
Yeah it's a great idea & i've been waiting for a bigger OSS LLM to arrive to do just that; As many use cases for embeddings are commercial it has to be an OSS one like you said, i.e. not llama
MPT-30B or Falcon seem like good candidates, but I think it's worth waiting for https://www.mosaicml.com/blog/introducing-ai2-olmo 🧐
Interesting, but arriving early 2024. That's waiting half a year, an eternity in this space. Then again, I guess it is relatively costly to produce sgpt embeddings the larger the model gets, right?
True it's quite far off 😅
Yeah the larger the model the more expensive - I think it may be worth adding a linear layer at the end to downscale the embedding size to e.g. 2048 for those very large models