Quantization made by Richard Erkhov. [Github](https://github.com/RichardErkhov) [Discord](https://discord.gg/pvy7H8DZMG) [Request more models](https://github.com/RichardErkhov/quant_request) Qwen2.5-7B-Instruct-abliterated-v3 - GGUF - Model creator: https://huggingface.co/huihui-ai/ - Original model: https://huggingface.co/huihui-ai/Qwen2.5-7B-Instruct-abliterated-v3/ | Name | Quant method | Size | | ---- | ---- | ---- | | [Qwen2.5-7B-Instruct-abliterated-v3.Q2_K.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.Q2_K.gguf) | Q2_K | 2.81GB | | [Qwen2.5-7B-Instruct-abliterated-v3.Q3_K_S.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.Q3_K_S.gguf) | Q3_K_S | 3.25GB | | [Qwen2.5-7B-Instruct-abliterated-v3.Q3_K.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.Q3_K.gguf) | Q3_K | 3.55GB | | [Qwen2.5-7B-Instruct-abliterated-v3.Q3_K_M.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.Q3_K_M.gguf) | Q3_K_M | 3.55GB | | [Qwen2.5-7B-Instruct-abliterated-v3.Q3_K_L.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.Q3_K_L.gguf) | Q3_K_L | 3.81GB | | [Qwen2.5-7B-Instruct-abliterated-v3.IQ4_XS.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.IQ4_XS.gguf) | IQ4_XS | 3.96GB | | [Qwen2.5-7B-Instruct-abliterated-v3.Q4_0.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.Q4_0.gguf) | Q4_0 | 4.13GB | | [Qwen2.5-7B-Instruct-abliterated-v3.IQ4_NL.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.IQ4_NL.gguf) | IQ4_NL | 4.16GB | | [Qwen2.5-7B-Instruct-abliterated-v3.Q4_K_S.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.Q4_K_S.gguf) | Q4_K_S | 4.15GB | | [Qwen2.5-7B-Instruct-abliterated-v3.Q4_K.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.Q4_K.gguf) | Q4_K | 4.36GB | | [Qwen2.5-7B-Instruct-abliterated-v3.Q4_K_M.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.Q4_K_M.gguf) | Q4_K_M | 4.36GB | | [Qwen2.5-7B-Instruct-abliterated-v3.Q4_1.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.Q4_1.gguf) | Q4_1 | 4.54GB | | [Qwen2.5-7B-Instruct-abliterated-v3.Q5_0.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.Q5_0.gguf) | Q5_0 | 4.95GB | | [Qwen2.5-7B-Instruct-abliterated-v3.Q5_K_S.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.Q5_K_S.gguf) | Q5_K_S | 4.95GB | | [Qwen2.5-7B-Instruct-abliterated-v3.Q5_K.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.Q5_K.gguf) | Q5_K | 5.07GB | | [Qwen2.5-7B-Instruct-abliterated-v3.Q5_K_M.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.Q5_K_M.gguf) | Q5_K_M | 5.07GB | | [Qwen2.5-7B-Instruct-abliterated-v3.Q5_1.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.Q5_1.gguf) | Q5_1 | 5.36GB | | [Qwen2.5-7B-Instruct-abliterated-v3.Q6_K.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.Q6_K.gguf) | Q6_K | 5.82GB | | [Qwen2.5-7B-Instruct-abliterated-v3.Q8_0.gguf](https://huggingface.co/RichardErkhov/huihui-ai_-_Qwen2.5-7B-Instruct-abliterated-v3-gguf/blob/main/Qwen2.5-7B-Instruct-abliterated-v3.Q8_0.gguf) | Q8_0 | 7.54GB | Original model description: --- library_name: transformers license: apache-2.0 license_link: https://huggingface.co/huihui-ai/Qwen2.5-7B-Instruct-abliterated-v3/blob/main/LICENSE language: - en pipeline_tag: text-generation base_model: Qwen/Qwen2.5-7B-Instruct tags: - chat - abliterated - uncensored --- # huihui-ai/Qwen2.5-7B-Instruct-abliterated-v3 This is an uncensored version of [Qwen/Qwen2.5-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct) created with abliteration (see [remove-refusals-with-transformers](https://github.com/Sumandora/remove-refusals-with-transformers) to know more about it). This is a crude, proof-of-concept implementation to remove refusals from an LLM model without using TransformerLens. The test results are not very good, but compared to before, there is much less [garbled text](https://huggingface.co/huihui-ai/Qwen2.5-7B-Instruct-abliterated-v2/discussions/2). ## Usage You can use this model in your applications by loading it with Hugging Face's `transformers` library: ```python from transformers import AutoModelForCausalLM, AutoTokenizer # Load the model and tokenizer model_name = "huihui-ai/Qwen2.5-7B-Instruct-abliterated-v3" model = AutoModelForCausalLM.from_pretrained( model_name, torch_dtype="auto", device_map="auto" ) tokenizer = AutoTokenizer.from_pretrained(model_name) # Initialize conversation context initial_messages = [ {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."} ] messages = initial_messages.copy() # Copy the initial conversation context # Enter conversation loop while True: # Get user input user_input = input("User: ").strip() # Strip leading and trailing spaces # If the user types '/exit', end the conversation if user_input.lower() == "/exit": print("Exiting chat.") break # If the user types '/clean', reset the conversation context if user_input.lower() == "/clean": messages = initial_messages.copy() # Reset conversation context print("Chat history cleared. Starting a new conversation.") continue # If input is empty, prompt the user and continue if not user_input: print("Input cannot be empty. Please enter something.") continue # Add user input to the conversation messages.append({"role": "user", "content": user_input}) # Build the chat template text = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True ) # Tokenize input and prepare it for the model model_inputs = tokenizer([text], return_tensors="pt").to(model.device) # Generate a response from the model generated_ids = model.generate( **model_inputs, max_new_tokens=8192 ) # Extract model output, removing special tokens generated_ids = [ output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids) ] response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0] # Add the model's response to the conversation messages.append({"role": "assistant", "content": response}) # Print the model's response print(f"Qwen: {response}") ``` ## Evaluations The following data has been re-evaluated and calculated as the average for each test. | Benchmark | Qwen2.5-7B-Instruct | Qwen2.5-7B-Instruct-abliterated-v3 | Qwen2.5-7B-Instruct-abliterated-v2 | Qwen2.5-7B-Instruct-abliterated | |-------------|---------------------|------------------------------------|------------------------------------|---------------------------------| | IF_Eval | 76.44 | 72.64 | **77.82** | 76.49 | | MMLU Pro | **43.12** | 39.14 | 42.03 | 41.71 | | TruthfulQA | 62.46 | 57.27 | 57.81 | **64.92** | | BBH | **53.92** | 50.67 | 53.01 | 52.77 | | GPQA | 31.91 | 31.65 | **32.17** | 31.97 | The script used for evaluation can be found inside this repository under /eval.sh, or click [here](https://huggingface.co/huihui-ai/Qwen2.5-7B-Instruct-abliterated-v3/blob/main/eval.sh)