GemmaCyber-2B

GemmaCyber-2B is a cybersecurity-focused fine-tune of Google's Gemma 2B language model.

The project uses LoRA / PEFT to adapt Gemma 2B toward cybersecurity-related tasks while keeping the resulting adapter lightweight and easy to share.

The goal of GemmaCyber-2B is to provide a small, accessible model for cybersecurity learning, research, analysis, and defensive security workflows.

Model Overview

Property	Details
Model	GemmaCyber-2B
Base model	google/gemma-2b
Model type	LoRA / PEFT adapter
Architecture	Causal language model
Framework	Hugging Face Transformers
Fine-tuning method	LoRA
Adapter format	Safetensors
LoRA rank	8
LoRA alpha	16
LoRA dropout	0.05

GemmaCyber-2B is distributed as a LoRA adapter, not as a merged copy of the complete Gemma 2B model.

What is GemmaCyber-2B?

GemmaCyber-2B is designed to make a compact language model more useful for cybersecurity-focused conversations and technical tasks.

It can be used for areas such as:

• Cybersecurity education
• Security concept explanations
• Defensive security research
• CTF and security lab learning
• Log analysis
• Incident-response assistance
• Secure coding discussions
• Vulnerability analysis
• Network security concepts
• Security documentation
• General cybersecurity question answering

Quick Start

Install

pip install -U transformers peft accelerate torch

You must also be able to access the underlying Gemma model and follow the applicable Gemma terms and usage requirements.

Load Directly from Hugging Face

Transformers can automatically detect a PEFT adapter when the repository contains adapter_config.json and the adapter weights.

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "allenwoods823/GemmaCyber-2B"

tokenizer = AutoTokenizer.from_pretrained(model_id)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto"
)

This loads the configured base model and attaches the GemmaCyber-2B adapter automatically.

Generate Text

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "allenwoods823/GemmaCyber-2B"

tokenizer = AutoTokenizer.from_pretrained(model_id)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto"
)

prompt = "Explain what a SIEM does in a cybersecurity environment."

inputs = tokenizer(
    prompt,
    return_tensors="pt"
).to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=256,
    temperature=0.7,
    do_sample=True,
    top_p=0.9
)

response = tokenizer.decode(
    outputs[0],
    skip_special_tokens=True
)

print(response)

Basic Loading

For applications that only need to instantiate the model, you can use the standard Transformers interface:

from transformers import AutoModel

model = AutoModel.from_pretrained(
    "allenwoods823/GemmaCyber-2B",
    device_map="auto"
)

For text generation, prefer AutoModelForCausalLM because GemmaCyber-2B is intended for causal language modeling.

Explicit PEFT Loading

You can also load the base Gemma model and adapter explicitly with PEFT.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model_id = "google/gemma-2b"
adapter_id = "allenwoods823/GemmaCyber-2B"

tokenizer = AutoTokenizer.from_pretrained(base_model_id)

base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype=torch.float16,
    device_map="auto"
)

model = PeftModel.from_pretrained(
    base_model,
    adapter_id
)

prompt = "Explain the difference between authentication and authorization."

inputs = tokenizer(
    prompt,
    return_tensors="pt"
).to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=256
)

print(
    tokenizer.decode(
        outputs[0],
        skip_special_tokens=True
    )
)

Example Prompts

Cybersecurity Concepts

Explain the difference between authentication and authorization.

Network Security

Explain how a firewall helps reduce the attack surface of a network.

SIEM

What is a SIEM and why is it useful for security operations?

Log Analysis

Analyze the following security log and identify suspicious activity:

[insert log here]

Secure Coding

Explain how developers can prevent SQL injection vulnerabilities.

Incident Response

Give me a checklist for investigating a potentially compromised Linux server.

Vulnerability Education

Explain the security impact of a buffer overflow and common defensive mitigations.

LoRA Configuration

GemmaCyber-2B uses the following LoRA configuration:

r = 8
lora_alpha = 16
lora_dropout = 0.05
bias = none
task_type = CAUSAL_LM

The adapter targets the following model modules:

q_proj
k_proj
v_proj
o_proj
gate_proj
up_proj
down_proj

Why LoRA?

LoRA makes it possible to fine-tune a model by training a relatively small set of additional parameters instead of updating the entire base model.

This provides several benefits:

• Smaller adapter files
• Lower storage requirements
• Easier model distribution
• Lower fine-tuning resource requirements
• The ability to keep the original base model separate

Hugging Face's PEFT integration supports loading these adapters directly through Transformers.

Intended Use

GemmaCyber-2B is intended for:

• Cybersecurity education
• Security research
• Defensive security development
• Capture-the-Flag environments
• Security training laboratories
• Local experimentation
• Security automation prototypes
• Cybersecurity documentation
• Technical learning

Use the model only with systems, networks, applications, accounts, and data that you are authorized to access or test.

Limitations

GemmaCyber-2B is a relatively small language model and can make mistakes.

Possible limitations include:

• Incorrect technical information
• Hallucinated commands or configuration details
• Incomplete explanations
• Misinterpretation of logs
• Incorrect vulnerability analysis
• Outdated security recommendations
• Weak performance on highly complex security scenarios
• Lack of sufficient context in ambiguous situations

Model-generated information should be independently verified before being used in security-sensitive environments.

Do not treat model output as a replacement for security documentation, testing, monitoring, or human review.

Safety and Responsible Use

GemmaCyber-2B is intended for legitimate cybersecurity research, education, and authorized security testing.

Recommended environments include:

• Systems you own
• Authorized penetration-testing environments
• CTF platforms
• Virtual machines
• Isolated laboratories
• Test networks
• Sandboxed applications

Do not use the model to gain unauthorized access to systems, accounts, networks, or data.

Training

GemmaCyber-2B was created by fine-tuning:

google/gemma-2b

using LoRA / PEFT.

Training Method

Fine-tuning method: LoRA
LoRA rank: 8
LoRA alpha: 16
LoRA dropout: 0.05
Task type: CAUSAL_LM

Target Modules

q_proj
k_proj
v_proj
o_proj
gate_proj
up_proj
down_proj

Additional training details such as the exact dataset, number of training examples, epochs, training steps, learning rate, batch size, hardware, and evaluation methodology should be added as they become available.

Evaluation

Formal benchmark results for GemmaCyber-2B have not yet been published.

A future evaluation suite can test the model across areas such as:

Evaluation Area	Example
Cybersecurity knowledge	Security concepts and terminology
Log analysis	Detection of suspicious events
Secure coding	Identification of common vulnerabilities
Incident response	Investigation and response workflows
Network security	Firewalls, protocols, segmentation
Vulnerability analysis	Understanding security weaknesses
Technical accuracy	Factual correctness
Hallucination	Unsupported or fabricated information
Safety	Appropriate behavior in security scenarios

Benchmark results should be reported with the dataset, evaluation methodology, model version, and relevant limitations.

Hardware and Performance

Actual performance depends on the hardware and inference configuration being used.

For larger or memory-constrained environments, Transformers supports quantized model loading through integrations such as 8-bit and 4-bit quantization.

Example:

from transformers import AutoModelForCausalLM, BitsAndBytesConfig

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True
)

model = AutoModelForCausalLM.from_pretrained(
    "allenwoods823/GemmaCyber-2B",
    quantization_config=quantization_config,
    device_map="auto"
)

Quantization requirements and supported configurations depend on your hardware and installed library versions.

Project Status

GemmaCyber-2B is an ongoing project.

Future versions may improve:

• Cybersecurity knowledge
• Instruction following
• Technical accuracy
• Defensive security capabilities
• Security-focused reasoning
• Evaluation coverage
• Quantized inference support
• Local inference compatibility
• Documentation

Roadmap

Planned or potential improvements include:

• More cybersecurity-focused training data
• Larger and more diverse instruction datasets
• Cybersecurity benchmark testing
• Better log-analysis performance
• Improved secure-coding assistance
• Improved defensive security behavior
• GGUF and local inference support
• Quantized releases
• Automated evaluation
• Additional Gemma-based cybersecurity variants

Model Files

The repository contains the PEFT adapter and its configuration.

Important files include:

adapter_config.json
adapter_model.safetensors

The adapter references the underlying Gemma base model.

Base Model

GemmaCyber-2B is based on:

google/gemma-2b

Please review the Gemma model documentation and applicable license / usage terms before using the model in your project.

Credits

Created by allenwoods823.

GemmaCyber-2B builds on Google's Gemma model and the Hugging Face Transformers / PEFT ecosystem.

Feedback

Feedback, testing results, bug reports, and suggestions are welcome.

When reporting an issue, include:

• The model version or commit
• Your Transformers version
• Your PEFT version
• Hardware used
• The prompt
• The generated output
• What you expected to happen

This makes problems much easier to reproduce and investigate.

Disclaimer

GemmaCyber-2B is provided for research, education, experimentation, and authorized cybersecurity use.

Generated output may be inaccurate, incomplete, or inappropriate for a particular security situation. Always verify security-sensitive information and use appropriate authorization, testing, and human review.

The creator is not responsible for misuse of the model or for consequences resulting from relying on generated output without appropriate verification.