HD-RL: Human Decision RL Policy (LoRA Adapter)

This repository contains a LoRA adapter trained using reinforcement learning framework for optimizing LLM explanations for human decision making.

The model is designed to analyze a given text and generate explanation hints/bullets outlining linguistic markers of AI or human authorship.

---

Model Details

• Base Model: Qwen/Qwen3.6-27B
• Adapter Type: LoRA (Low-Rank Adaptation)
• Task: Generating strict, flat-list linguistic explanation hints based on specific formatting templates.

---

Output Constraints & Format

When prompted correctly, the policy strictly follows these constraints:

1. Starts the response immediately with a bulleted list (2 to 6 bullet points).
2. Does not include an introduction, final verdict, or summary.
3. Keeps it a flat list (no sub-headings).
4. Follows precise templates:
   • - "Look for [linguistic category or type of phrasing]."
   • - "Look for [linguistic category or type of phrasing] like '[short example]'."
5. Avoids bolding, italics, or other special Markdown formatting.

---

How to Run

To use this model for inference, you can load the base model via transformers and apply the LoRA weights using peft.

Prerequisites

Make sure you have the required libraries installed:

pip install torch transformers peft accelerate

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# 1. Define model identifiers
BASE_MODEL = "Qwen/Qwen3.6-27B"
ADAPTER_MODEL = "Tizzzzy/HD-RL"  # Your Hugging Face repository

# 2. Load tokenizer
print("Loading tokenizer...")
tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL, trust_remote_code=True)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

# 3. Load base model in bfloat16 with automatic device mapping
print("Loading base model...")
base_model = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

# 4. Load and attach the LoRA adapter
print("Loading LoRA adapter...")
model = PeftModel.from_pretrained(base_model, ADAPTER_MODEL)
model.eval()

# 5. Define your prompt template and constraints
POLICY_INSTRUCTION = """Task: Analyze the provided text for linguistic markers of AI or human authorship.

Constraints:
1. Start the response IMMEDIATELY with the bulleted list.
2. Do NOT provide an introduction, a final verdict, or a summary.
3. Do NOT group the bullet points into sub-headings (keep it a flat list).
4. Every bullet point MUST follow one of these two exact templates:
- "Look for [linguistic category or type of phrasing]."
- "Look for [linguistic category or type of phrasing] like '[short example]'."
5. When providing examples, you may list multiple words separated by commas (e.g., like "word 1", "word 2").
6. Do NOT use bolding, italics, or any special Markdown formatting.
7. Limit the output to exactly 2 to 6 bullet points.

Text:
{text}

Can you provide an explanation that tells the reader what to focus on?

Explanation:
"""

# 6. Format input text
sample_text = "Your sample text goes here to analyze its human or AI linguistic features."
messages = [{"role": "user", "content": POLICY_INSTRUCTION.format(text=sample_text)}]

# Apply chat template (disabling thinking mode if applicable)
try:
    prompt = tokenizer.apply_chat_template(
        messages, tokenize=False, add_generation_prompt=True, enable_thinking=False
    )
except TypeError:
    prompt = tokenizer.apply_chat_template(
        messages, tokenize=False, add_generation_prompt=True, chat_template_kwargs={"enable_thinking": False}
    )

# 7. Tokenize and generate
inputs = tokenizer(prompt, return_tensors="pt", truncation=True, max_length=4096).to(model.device)

with torch.no_grad():
    output_ids = model.generate(
        **inputs,
        max_new_tokens=512,
        temperature=1.0,
        top_p=1.0,
        repetition_penalty=1.1,
        do_sample=True,
        pad_token_id=tokenizer.eos_token_id,
    )

# Extract generated tokens (strip out the input prompt)
prompt_len = inputs["input_ids"].shape[1]
generated_ids = output_ids[0, prompt_len:]
explanation = tokenizer.decode(generated_ids, skip_special_tokens=True).strip()

print("\n--- Generated Explanation ---")
print(explanation)