GPT (AIhomeJP/home1)

GPT is a lightweight GPT-style causal language model with a recursive refinement mechanism built on top of standard Transformer blocks.

This model is implemented with custom Hugging Face-compatible code and requires trust_remote_code=True when loading.

---

Model Description

GPT extends a standard decoder-only Transformer by adding a recursive refinement stage:

• Standard Transformer layers process the sequence
• A refinement block iteratively improves hidden states
• A gating mechanism controls residual blending

Core update equation

$$h_{t+1}=g(h_t)\cdot \mathrm{B}\mathrm{l}\mathrm{o}\mathrm{c}\mathrm{k}(h_t)+(1-g(h_t))\cdot h_{t}h\_\{t+1\}\; =\; g(h\_t)\; \backslash cdot\; \backslash mathrm\{Block\}(h\_t)\; +\; (1\; -\; g(h\_t))\; \backslash cdot\; h\_t$$

---

Model Details

Parameter	Value
Architecture	Decoder-only Transformer
Layers	4
Hidden size	256
Heads	4
Context length	256
Recursive steps	2

---

Usage

Load Model

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "AIhomeJP/home1",
    trust_remote_code=True
)

tokenizer = AutoTokenizer.from_pretrained("AIhomeJP/home1")

---

Generate Text

import torch

input_ids = tokenizer("Hello", return_tensors="pt").input_ids

# モデルを評価モードに設定
model.eval()
curr_ids = input_ids

with torch.no_grad():
    for _ in range(10):
        # 辞書形式で出力が返るため、ブラケットでアクセスします
        outputs = model(curr_ids)
        logits = outputs['logits'][:, -1, :]
        
        # NaNエラーを回避するためargmaxを使用
        next_id = torch.argmax(logits, dim=-1, keepdim=True)
        curr_ids = torch.cat([curr_ids, next_id], dim=1)

print(tokenizer.decode(curr_ids[0]))

---

Training

This model was trained using a supervised fine-tuning (SFT) pipeline with a standard causal language modeling objective:

$$\mathcal{L}=\mathrm{C}\mathrm{r}\mathrm{o}\mathrm{s}\mathrm{s}\mathrm{E}\mathrm{n}\mathrm{t}\mathrm{r}\mathrm{o}\mathrm{p}\mathrm{y}(\text{logits},\text{labels})\backslash mathcal\{L\}\; =\; \backslash mathrm\{CrossEntropy\}(\backslash text\{logits\},\; \backslash text\{labels\})$$

---

Implementation Notes

Weight Tying

The model shares weights between:

• token_emb.weight
• lm_head.weight

This is explicitly defined via:

_tied_weights_keys = ["lm_head.weight"]

---

Limitations

• Small model size (limited generalization)
• No KV-cache (generation latency is higher)
• No advanced attention optimizations
• Experimental architecture

---

Requirements

pip install torch transformers

---

License

MIT License

---

Author

• summer