Impressive Context Awareness!
#8
by
akumaburn
- opened
I've been testing several dozen models for context awareness and yours has performed exceedingly well.
Congratulations!
It would be interesting to see how this performs if fine tuned further..
Perhaps with the OpenOrca (https://huggingface.co/datasets/Open-Orca/OpenOrca) dataset?
I wonder what 1 epoch worth of training with that would do this model.
Thank you, I’d be willing to take all examples nearing 2048 tokens, for the continued pre training, i’m still working on it. I have a decent dataset for getting the perplexity down, so far the lowest I’ve gotten is 25 (for reference v.02 gets 75 on perplexity) ; and I was looking for some more variation in the datatypes, I think this is good.
Thank you 👍❤️🤗
Vezora
changed discussion status to
closed