README.md

metadata

base_model: unsloth/llama-3.2-1b-instruct-unsloth-bnb-4bit
library_name: peft
pipeline_tag: text-generation
tags:
  - base_model:adapter:unsloth/llama-3.2-1b-instruct-unsloth-bnb-4bit
  - lora
  - sft
  - transformers
  - trl
  - unsloth
license: mit
datasets:
  - ServiceNow-AI/R1-Distill-SFT
language:
  - en

Model Card for Model ID

• Its a very simple model for text generation built on top of Llama3.2-1B.

• It is very lightweight and can be inferenced on a CPU with 4 gb RAM.

• Developed by: findthehead

Framework versions

• PEFT 0.17.1

Inference Code

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

model_name = "Prachir-AI/Thinkmini"
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Create a BitsAndBytesConfig to enable 4-bit loading
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True, # Enable 4-bit loading as intended for this model
    bnb_4bit_quant_type="nf4", # This is a common default for 4-bit models
    bnb_4bit_compute_dtype=torch.bfloat16, # Use bfloat16 for computation
    bnb_4bit_use_double_quant=True, # Often used with nf4
)

# Load the model with the configured 4-bit quantization
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    quantization_config=bnb_config,
    torch_dtype=torch.bfloat16 # Ensure the model itself is loaded with bfloat16 dtypes where applicable
)

inputs = tokenizer("How do you plan for a full pentest of a web application?", return_tensors="pt").to('cuda')
  # inference mode

output_ids = model.generate(
    **inputs,
    max_new_tokens=500,
    temperature=0.7,
    top_p=0.9
)

print(tokenizer.decode(output_ids[0], skip_special_tokens=True))