THUDM

/

cogvlm2-llama3-chat-19B-tgi

like 3

Follow

Knowledge Engineering Group (KEG) & Data Mining at Tsinghua University 1,129

that makes both of us. Any solution yet?

We have tried both TGI and LM Deploy for CogVLM2. It's still not working on TGI. It works on LM Deploy in full precision(you need a > 45GB card for this), but not in 4 or 8 bits.

I'm encountering an issue when attempting to run the cogvlm2 model using TGI Docker image. Below is the script I've been using:

model=THUDM/cogvlm2-llama3-chat-19B-tgi
volume=$PWD/data
docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data
ghcr.io/huggingface/text-generation-inference:2.0.4
--model-id $model --trust-remote-code

When I execute this, I receive the following error:
ValueError: Unsupported model type cogvlm2

I've already looked into the text generation inference documentation for running non-core models, but I'm still facing the same issue.
Could anyone help me understand why this error occurs and how I might fix it? Is there a specific configuration or version that supports cogvlm2 model types?

Thanks for your assistance!

Facing same issue