--- base_model: Qwen/Qwen3.5-2B library_name: transformers pipeline_tag: text-generation license: apache-2.0 tags: - qwen3.5 - lora - peft - vllm - document-processing --- # Shrew LoRA Adapters LoRA adapters for [Qwen/Qwen3.5-2B](https://huggingface.co/Qwen/Qwen3.5-2B) fine-tuned for structured extraction as part of a production RAG application. These power [Shrew's](https://github.com/btbtyler09/shrew-server) structured extraction pipeline. ## Adapters | Adapter | Tasks | LoRA rank / alpha | Status | |---|---|---|---| | **`doc_processing/`** | `extract_metadata`, `semantic_chunk`, `summarize_document` (3 tasks, single adapter) | r128 / α256 | **Recommended** — unified, supersedes the 3 per-task adapters below | | `extract_metadata/` | Extract structured metadata (title, authors, dates, document type) | r32 / α64 | Superseded by `doc_processing/` | | `summarize_document/` | Generate document summaries | r32 / α64 | Superseded by `doc_processing/` | | `semantic_chunk/` | Split documents into semantically coherent sections | r128 / α256 | Superseded by `doc_processing/` | The unified `doc_processing/` adapter routes by **system prompt** — the prompt is just the task name (`extract_metadata`, `semantic_chunk`, or `summarize_document`). One adapter, three tasks. ### `doc_processing/` metrics | | | |---|---| | Base model | `Qwen/Qwen3.5-2B` | | LoRA rank / alpha | 128 / 256 | | Adapter size | ~256 MB (f16) | | Training corpus | ~103k examples across 3 tasks | | Epochs | 3 | | Final eval loss | 0.7103 | | Hardware | 4× AMD MI100 (gfx908), bf16, DeepSpeed ZeRO-2 | ## Usage The unified `doc_processing/` adapter routes by system prompt. Same adapter, different task per call. ```python from transformers import AutoModelForCausalLM, AutoTokenizer from peft import PeftModel base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-2B") tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-2B") model = PeftModel.from_pretrained(base, "btbtyler09/shrew-2b", subfolder="doc_processing") messages = [ {"role": "system", "content": "extract_metadata"}, # or "semantic_chunk", "summarize_document" {"role": "user", "content": document_text}, ] prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, enable_thinking=False) ``` `extract_metadata` and `semantic_chunk` produce JSON; `summarize_document` produces prose. ### llama.cpp (GGUF) GGUF versions can be used as LoRA adapters with `llama-cli` / `llama-server`: ```bash llama-server -m Qwen3.5-2B.gguf --lora doc_processing.gguf ``` ### vLLM vLLM loads Qwen3.5 as `Qwen3_5ForConditionalGeneration` (a VLM class), which nests the language model under a `language_model.` prefix. The adapters here are saved in standard PEFT format (trained with `AutoModelForCausalLM`), so the weight keys must be renamed before serving with vLLM's `--enable-lora`. Without the rename, vLLM silently zeros all LoRA weights with no error. Apply the rename with `fix_lora_keys.py` from the [shrew](https://github.com/btbtyler09/shrew) repo: ```bash python fix_lora_keys.py path/to/adapter vllm serve Qwen/Qwen3.5-2B \ --enable-lora \ --lora-modules doc_processing=path/to/adapter \ --max-lora-rank 128 \ --max-loras 1 ``` Note `--max-lora-rank 128` for the unified adapter (vLLM's default of 16 is too low). ## Sampling parameters Use Qwen 3.5 instruct-general parameters with `enable_thinking=False`: | param | value | |---|---| | temperature | 0.7 | | top_p | 0.8 | | top_k | 20 | ## License Same as base model ([Qwen/Qwen3.5-2B](https://huggingface.co/Qwen/Qwen3.5-2B)).