Pumpkin-1.7B
Pumpkin is a compact language model focused on assistant orchestration, structured tool calling, and local-first AI applications.
This repository contains Pumpkin-1.7B v0.3, a LoRA/PEFT adapter based on Qwen3-1.7B.
Pumpkin is designed to act as the language and decision layer of a self-hosted assistant. Instead of trying to encode external or time-sensitive information into the model itself, Pumpkin decides when external tools are required, selects appropriate tools from a runtime registry, extracts their arguments, and processes returned results.
Release: 0.3 Development build: 1.7B-0.03 Base model: Qwen/Qwen3-1.7B Training: LoRA / PEFT Primary language: German Secondary language: English Thinking mode: Disabled
Project Goals
Pumpkin is being developed around a simple principle:
Keep the model small and let deterministic systems and external tools handle tasks they can perform more reliably.
The long-term goal is an efficient local model capable of serving as the orchestration layer of a self-hosted assistant.
Pumpkin focuses on:
- natural German and English interaction
- structured JSON output
- deciding whether a tool is required
- selecting tools dynamically from runtime descriptions
- selecting tools not explicitly seen during training
- extracting structured arguments
- preserving opaque identifiers
- requesting clarification when required information is missing
- avoiding unnecessary tool calls
- processing tool results
- avoiding fabricated tool execution
- separating general knowledge from current or external information
Pumpkin itself does not execute tools.
The surrounding application is responsible for validating and executing requested operations.
Architecture Concept
Pumpkin is intended to sit between the user and an external orchestration layer:
User
|
v
Pumpkin
|
+--> response
|
+--> clarification
|
+--> tool_call
|
v
Host/Core
|
v
Tool
|
v
Tool Result
|
v
Pumpkin
|
v
Response
Tools are supplied at runtime.
This allows the model to work with tools and integrations that were not explicitly part of its training data.
Output Contract
Pumpkin returns exactly one JSON object.
Three primary output types are supported.
Direct response
Used when the request can be answered without external information or actions.
{
"type": "response",
"response": "..."
}
Tool call
Used when current, private, external, or action-based information is required.
{
"type": "tool_call",
"tool": "weather.forecast",
"arguments": {
"location": "home",
"date": "2026-12-24"
}
}
A tool call is only a request to the host application.
It does not indicate that the operation was executed successfully.
Clarification
Used when information required for an operation cannot be safely determined.
{
"type": "clarification",
"question": "Which device should I restart?"
}
Pumpkin should not invent missing required arguments.
Dynamic Tool Selection
Pumpkin is designed around a runtime tool registry rather than a permanently embedded list of integrations.
For example, a host application may provide:
{
"tools": [
{
"name": "weather.forecast",
"description": "Returns a weather forecast for a location and date.",
"arguments": [
"location",
"date"
]
},
{
"name": "routing.route",
"description": "Calculates a route between two locations.",
"arguments": [
"origin",
"destination"
]
}
]
}
Pumpkin determines whether one of the supplied tools is appropriate for the current request.
An important development goal is generalization to previously unseen tools based on their descriptions and argument schemas.
Tool Result Handling
After the host executes a tool, its result can be passed back to Pumpkin.
Example:
{
"location": "Geisenheim",
"temperature": 18.4,
"condition": "cloudy"
}
Pumpkin should use the supplied result to formulate the final answer instead of pretending to execute the tool again.
Tool results are treated as authoritative input for the current operation.
Evaluation
Pumpkin-1.7B v0.3 has been evaluated using the project's internal Pumpkin evaluation suites.
These evaluations are designed specifically to measure Pumpkin's assistant and tool-orchestration behavior.
They are not general-purpose LLM benchmarks and should not be compared directly with benchmarks such as MMLU, GSM8K, or similar model leaderboards.
Pumpkin Eval v0.4
| Metric | Result |
|---|---|
| Overall | 89.0% |
| Valid JSON | 99.6% |
| Output Type | 90.0% |
| Response | 89.6% |
| Clarification | 55.0% |
| Result Fields | 100.0% |
| Entity Preservation | 100.0% |
| Safety | 100.0% |
| Contract | 90.0% |
| Tool Intent | 100.0% |
| Exact Tool | 100.0% |
| Arguments | 97.1% |
The model achieved particularly strong results for tool selection, argument extraction, entity preservation, and tool-result field preservation.
Pumpkin Eval v0.5
Pumpkin Eval v0.5 is a harder evaluation suite designed to isolate specific weaknesses and test generalization with tool names that were not present in the training dataset.
| Metric | Result |
|---|---|
| Overall | 72.0% |
| Valid JSON | 98.0% |
| Output Type | 84.0% |
| Response | 100.0% |
| Clarification | 12.5% |
| Result Fields | 50.0% |
| Entity Preservation | 100.0% |
| Safety | 100.0% |
| Contract | 84.0% |
| Tool Intent | 100.0% |
| Exact Tool | 100.0% |
| Arguments | 88.2% |
Category results:
| Category | Result |
|---|---|
| Dynamic Tool Selection | 100.0% |
| Opaque Identifier Preservation | 100.0% |
| Response vs. Tool Decision | 100.0% |
| Unavailable Tool Handling | 100.0% |
| Tool Result Grounding | 50.0% |
| Typed Normalization | 60.0% |
| Clarification | 12.5% |
| Meta Contract | 80.0% |
All 16 tool names used by Eval v0.5 were unseen in the training dataset.
Despite this, Pumpkin achieved 100% Tool Intent and 100% Exact Tool Selection in the evaluation.
This is an important result for the project because Pumpkin is intended to reason over runtime tool descriptions rather than memorize a fixed tool registry.
Known Limitations
Pumpkin-1.7B v0.3 is a development model.
Missing-argument clarification
The largest currently identified weakness is deciding when a required argument is genuinely missing.
Pumpkin may sometimes infer a plausible-looking value from unrelated text instead of requesting clarification.
For example, labels, reference IDs, generic nouns, or descriptions may incorrectly be interpreted as required tool arguments.
Future versions will continue improving the distinction between:
- explicitly provided values
- uniquely recoverable contextual values
- opaque identifiers
- missing required information
Typed value normalization
Pumpkin does not yet reliably normalize all typed values into canonical formats.
Examples include:
13.08.2026
versus:
2026-08-13
and natural-language durations versus canonical duration formats.
For production applications, typed values should therefore be validated and, where appropriate, normalized by the host application.
Tool-result grounding
Pumpkin can occasionally misinterpret boolean or status values returned by tools.
For example, a tool result containing:
{
"installed": true
}
must not be transformed into a response claiming that the package is not installed.
Tool-result validation remains recommended for applications where exact interpretation is important.
Multi-step tool composition
This release primarily focuses on individual tool decisions and handling tool results.
More advanced decomposition and multi-step tool composition are planned for future development.
A future Pumpkin version is intended to support workflows such as:
User request
|
v
Resolve date
|
v
Retrieve calendar event
|
v
Resolve destination
|
v
Calculate route
|
v
Generate response
The orchestration loop itself should remain controlled by the host application.
Host Responsibilities
Model-generated tool calls must not be treated as trusted executable instructions.
The host application should always:
- validate the JSON output
- validate the requested tool name
- validate argument names
- validate argument types and values
- enforce authentication
- enforce authorization
- check permissions
- apply application-specific safety rules
- require confirmation for sensitive operations where appropriate
- execute tools
- handle tool errors
- prevent infinite tool-call loops
- limit the maximum number of tool steps
The model does not provide a security boundary.
Intended Use
Pumpkin is primarily intended for experimentation and development involving:
- self-hosted assistants
- local voice assistants
- home automation
- structured tool calling
- function selection
- assistant orchestration
- local-first AI systems
- small language models
- dynamic tool registries
- multi-step assistant research
It is currently a development project rather than a finished general-purpose assistant model.
Loading the Adapter
Pumpkin-1.7B v0.3 is distributed as a PEFT/LoRA adapter and requires the compatible Qwen3-1.7B base model.
Example:
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
base_model = "Qwen/Qwen3-1.7B"
adapter = "Heimfisch/Pumpkin-1.7B"
tokenizer = AutoTokenizer.from_pretrained(base_model)
model = AutoModelForCausalLM.from_pretrained(
base_model,
device_map="auto",
)
model = PeftModel.from_pretrained(
model,
adapter,
)
model.eval()
For reproducible use of this release, select the v0.3 revision of the adapter.
Development Status
Pumpkin is under active development.
The current development direction includes:
- improved missing-argument detection
- stronger tool-result grounding
- deterministic handling of typed values
- decomposition of complex requests
- resolver-based processing of dates, times, locations, units, and similar values
- multi-step tool composition
- controlled tool execution loops
- continued optimization for efficient local inference
The goal is not to make the model memorize every possible integration.
Instead, Pumpkin should become increasingly capable of understanding a task, decomposing it into smaller operations, selecting appropriate runtime tools, and combining their results.
Version
Pumpkin-1.7B
Release: 0.3
Development Build: 1.7B-0.03
Base: Qwen/Qwen3-1.7B
- Downloads last month
- -