TheBloke

/

Mistral-7B-Instruct-v0.1-AWQ

like 35

Please make sure you've updated Transformers and AutoAWQ to the required versions, as shown in the README:

Since I wrote that, AutoAWQ 0.1.1 released, so you can install AutoAWQ normally with :pip3 install autoawq (but make sure it's updated to 0.1.1)

Your issue looks like you're not running Transformers from Github, so please install the commit mentioned, or newer.

Hi @TheBloke , thanks for the quick reply!
I totally missed these instructions in the ReadME, I have updated the Transformers package to the commit.

However, there seems to be no 0.1.1 release for AutoAWQ and the installation for latest AutoAWQ commits seem buggy:

Oh sorry, I guess he only bumped the version to 0.1.1 in Github, and didn't do a release yet.

Here's how I install it:

git clone https://github.com/casper-hansen/AutoAWQ && \
    cd AutoAWQ && \
    git checkout v0.1.1 && \
    pip3 install .

Give that a go

I just realised that doesn't work either! As he's not tagged v 0.1.1 yet.

I'm trying this now:

git clone https://github.com/casper-hansen/AutoAWQ && \
    cd AutoAWQ && \
    git checkout 20cd798180785f297148c522481fab86950515fe && \
    pip3 install .

Seems like the package has issues with installation:

Try running pip3 install -v . to see more about what's happening.

I just built that commit in my Docker with no problems, so I think the package is fine - maybe an issue on your system.

What GPU are you using? AWQ requires an Ampere GPU or later, with compute capability 80 and above.

The command outputs this log.
Running the Colab session with a V100:

Yeah V100 is too old to support AWQ. If you want to use Google Colab you'll need to use an A100 if you want to use AWQ. Or use a different provider, like Runpod - they have many GPUs that would work, eg 3090, 4090, A4000, A4500, A5000, A6000, and many more.

@TheBloke does this model work with text-generation-webui?

Don't know for sure, but my guess would be no, as they likely need to update for Mistral support. Definitely doesn't work with vLLM, where the AWQ is Llama only at the moment.

AWQ v0.1.1 has now been released.

You can test out the AutoAWQ PR in ooba here: https://github.com/oobabooga/text-generation-webui/pull/3999.

Just make sure that you install autoawq after you have installed the PR: pip install autoawq==0.1.1.

I am still getting this error

KeyError: 'mistral'

using AWQ v0.1.1

You haven't updated Transformers - please see details in the README

Apologies for delay in reply, it was taking too long to load in the runtime that day.
It works now after updating the packages, thanks a lot for your help and all the great work for the community @TheBloke @casperhansen