How do I fit my knowledge base into this model?
#6
by
xyerr
- opened
I wanted to let SELF-RAG search for answers in my own knowledge base
You need to finetune your model with synthetic textbooks and that will never as good as pre-training your model, which is very expensive.
The idea is to finetune with a combination of empty prompts and generated instructions towards your data (summarization, q&a...).
A model like gpt4 generates the questions or tasks around your data.
According to the model card this is what has been done with the base model of this one SciPhi-Mistral-7B-32k, probably @emrgnt-cmplxty can give you some insight