where is opensource-code?
HI, I see some huggingface-transformer-like code in Model card, but I don't find Qwen2_5OmniModel in huggingface/transformers
Same issue, I want to see how process_mm_info works
https://github.com/huggingface/transformers/pull/36752 Bro could check this branch if you are in hurry @xubin-bruce
LOL..
yes it in the pull !!
There are still some issues with the pull !
but i do expect i will be solved soon ! check out the guy ! ----- BakerBunker on Github !
they did have the docs But i suspect these docs are very light as they have mainly only added the inference samples and not the finetuning setup or the configs ! for the audioFeatureextractor or the pre processor etc !
but i suspect it will be simular to the qwen audio setup and the onevision set up ... Quite tricky to instatiate a new model or model from pretrained components :)
being a new potential : Standard for models ..... Im sure they will not do the great docs ! .. Especially being chinese they often do not do great documentation as they seem to lapse in this department !
even the llava models were tricky to create !
Also as we get to the qwen models in particular they actually hijacked the LLAVA models ! Leaving the llava models hangiong ( still not implemented in llama cpp ! ) in favour of the qwen versions !
The problem is being forced to have a qwen model ! ---- and Qwen model Components ! ... the llava models do permit you to use a mistral or even a llama base modela s a prettrained llm component ! ... but the qwen models do not ! the jouirney is very long !
as you will need to make the base qwen model with your llama ( after it is ot the same as new gates have been added (mlp) these gates throw all your pretraining off ! ... but creating the llava model your data is retained somewhat after allignment dataset you will find your moel converging quite easy where as the qwen diverges !
Personally i love the deepseek etc but i also love my own pretrained model as it also contains very specific data to my self and family and business ! ... which has been trained over the 2 years with most paradigms ! ... So now i will be forced to use my own model as a agent and this model as a Dumb( orcestrator model ) ... not nice ! As it is all about tranfer learning and merging abilitys !
just check out the new docs !
Its a pity you cannot pull merge requests into your own fork!