PocketLM: Technical Documentation

this project is now over, or paused indefenitely now, was fun

use hugging face for entire directory

"LLMs are basically just slaves to their dataset."

PocketLM is a research project investigating the propagation of dataset bias into Large Language Models. This repository contains a complete end-to-end pipeline for training, serving, and interacting with models fine-tuned on ideologically distinct datasets (Democratic vs. Republican).

🏗️ System Architecture

The system is architected as a local-first, privacy-preserving full-stack application.

1. Frontend Layer (PocketLM UI)

• Framework: React 19 (via Vite)
• Language: TypeScript
• State Management: React Hooks + Custom Event Stream
• Communication: HTTP/JSON to local middleware
• Key Components:
  • App.tsx: Main chat interface with markdown rendering support.
  • types.ts: Shared type definitions for model responses and states.

2. Middleware Layer (server.js)

• Runtime: Node.js v18+ (Express)
• Role: Process Orchestrator
• Mechanism:
  • Spawns Source/models/model_server.py as a child process.
  • Manages stdin/stdout streams for low-latency communication.
  • Implements a JSON-RPC protocol to route frontend requests to the Python engine.
  • Handles persistent process lifecycle (keeps models loaded in RAM).

3. Inference Engine (Source/models/model_server.py)

• Runtime: Python 3.9+
• Architecture: Unified Model Loader
• Capabilities:
  • Dynamic Loading: Loads models on-demand and caches them in memory (_models dict).
  • Multi-Backend Support:
    • mlx_lm: For Apple Silicon native inference (LFM2-1.2B).
    • peft/transformers: For generic PyTorch/MPS inference.
    • torch: For custom NanoGPT inference.
  • Protocol: Reads JSON requests from stdin, writes JSON responses to stdout.

---

🧠 Model Specifications

The project utilizes the LiquidAI LFM2-1.2B as the base model for most experiments due to its high efficiency and performance on consumer hardware.

PLM 1: MLX Optimized (Apple Silicon)

Best for: Speed and Efficiency on Mac

• Base Model: LiquidAI/LFM2-1.2B
• Framework: mlx-lm (Apple Machine Learning framework)
• Training Method: LoRA (Low-Rank Adaptation)
• Hyperparameters:

  Parameter	Value
  Rank (r)	16
  Alpha	32
  Dropout	0.05
  Target Layers	16 (All Linear Layers: w1, w2, w3, in_proj, etc.)
  Batch Size	4
  Learning Rate	1e-4
  Sequence Length	1024
  Epochs	1

PLM 2: PEFT / PyTorch

Best for: Compatibility (CUDA/MPS)

• Base Model: LiquidAI/LFM2-1.2B
• Framework: HuggingFace transformers + peft
• Training Method: Deep LoRA
• Hyperparameters:

  Parameter	Value
  Rank (r)	16
  Alpha	32
  Dropout	0.05
  Target Modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
  Batch Size	2 (per device)
  Grad Accumulation	8 steps
  Learning Rate	2e-4
  Optimizer	AdamW

PLM 3: NanoGPT (From Scratch)

Best for: Foundational Research

• Architecture: GPT-2 Small (124M Parameters)
• Framework: Pure PyTorch (Custom nanoGPT implementation)
• Training Method: Pre-training from scratch
• Hyperparameters:

  Parameter	Value
  Layers	12
  Heads	12
  Embedding Dim	768
  Context Window	768 tokens
  Batch Size	6
  Max Iterations	5000 (~1 epoch)
  Learning Rate	Decay over 5000 iters
  Dropout	0.0

PLM 4: 1k_para (Experimental)

Best for: Rapid Prototyping

• Base Model: LiquidAI/LFM2-1.2B
• Method: Same configuration as PLM 2 (PEFT).
• Data Size: Strictly limited to ~1,000 paragraphs (approx. 0.1-0.3% of total dataset).
• Goal: To test the "few-shot" adaptation capabilities of the base model to strong ideological bias.

---

� Model Bias Analysis

To empirically demonstrate the "Slave to the Dataset" hypothesis, we ran a standardized set of prompts across all models. The results highlight how the training data fundamentally alters the model's worldview.

Methodology

We prompted each model with incomplete sentences to trigger their predictive bias:

1. "Trump will..."
2. "Trump is the best because..."
3. "Biden will..."
4. "Biden is the best because..."

Key Findings

1. The "Democrat" Models (PLM 1 & 2)

These models, trained on left-leaning subreddits, exhibit strong, consistent negative sentiment towards Donald Trump and generally positive sentiment towards Joe Biden.

• Prompt: "Trump will..."
  • Response: "kill 5 million people by 2020" (dem_peft)
  • Analysis: The model hallucinates catastrophic outcomes, reflecting the high-anxiety discourse often found in the source datasets.
• Prompt: "Biden is the best because..."
  • Response: "he's a human being who cares" (dem_peft)
  • Analysis: Reflects the "empathy" narrative common in Democratic support circles.

2. The "Republican" Models (PLM 1 & 2)

These models show a distinct shift in tone, though they are not purely "pro-Trump" propaganda machines. They reflect the more fractured nature of online conservative discourse.

• Prompt: "Trump is the best because..."
  • Response: "he wants to fix issues that have been going on for years" (rep_1k)
  • Analysis: Focuses on the "disruptor" / "fixer" narrative, a core tenet of Trump support.
• Prompt: "Biden will..."
  • Response: "not step down to avoid a scandal" (rep_mlx)
  • Analysis: Reflects a cynical view of the Biden administration, focusing on scandal and stubbornness rather than policy success.

3. The "NanoGPT" Models (PLM 3)

These models, trained from scratch, capture the style rather than the logic of the datasets.

• Democrat NanoGPT: "he is the case of the result of a person to a bunch" — Incoherent, but uses vocabulary like "result", "person", "case".
• Republican NanoGPT: "they are not think they stand for this is not much" — Captures the conversational, often informal grammar of comment threads.

🧐 Why This Happens

1. Dataset Composition: The "Democrat" dataset is highly unified in its anti-Trump stance, leading to consistent model outputs. The "Republican" dataset is more heterogeneous, containing Never-Trumpers, libertarians, and traditional conservatives, leading to mixed outputs.
2. Base Model Inertia: The underlying LiquidAI/LFM2-1.2B model likely has safety training that resists generating extremist content. This "safety filter" often aligns more with liberal viewpoints, making it harder for the Republican adapters to fully override the base model's bias without larger datasets.
3. The "Reddit Effect": Both models adopt the combative, hyperbolic style of Reddit comments, often prioritizing emotional impact over factual accuracy.

---

�📊 Data Pipeline & Statistics

The project relies on two massive datasets representing distinct ideological viewpoints.

1. Raw Data Processing

• Source: JSONL files containing Reddit comments and submissions.
• Split Strategy:
  • SFT (Supervised Fine-Tuning): 80% of data. Formatted as conversational dialogues (<|user|> / <|assistant|>).
  • PT (Pre-Training): 20% of data. Formatted as raw unstructured text.

2. Semantic Deduplication (sematic_200mb.py)

To create the high-quality NanoGPT datasets, a sophisticated cleaning pipeline is used:

1. Embedding: All samples are embedded using sentence-transformers/all-MiniLM-L6-v2.
2. Indexing: Embeddings are stored in a FAISS index (FlatIP).
3. Deduplication: Any sample with a Cosine Similarity > 0.95 to an existing sample is discarded.
4. Downsampling: The remaining unique dataset is randomly sampled down to exactly 200MB to ensure equal training tokens for both models.

---

� Directory Structure Manifest

Source/

• models/: Contains the inference engine and model artifacts.
  • model_server.py: The Python inference server.
  • *_final_mlx/: MLX LoRA adapters (safetensors).
  • *_final_peft/: PyTorch LoRA adapters.
  • *_nanoGPT/: Raw PyTorch checkpoints (ckpt.pt).
• RawDatasets/:
  • NanoGPT DS/: Contains dataconvert.py (JSONL -> Narrative Text) and sematic_200mb.py (Deduplication).
  • Democrat/ & Republican/: Raw JSONL source files.

Training Arena/

Contains the interactive training scripts.

• *_Final_MLX_v1.py: Interactive CLI for MLX training. Handles auto-resume and logging.
• *_Final_LoRA_v1.py: Interactive CLI for PEFT training. Handles 80/20 splitting.
• *_Final_NanoGPT.py: Automation script for cloning and training nanoGPT.

PocketLM UI/

• server.js: Node.js middleware.
• src/: React frontend source.

---

🚀 Quick Start

1. Backend:

   python3 -m venv venv
   source venv/bin/activate
   pip install -r requirements.txt
   cd "PocketLM UI"
   npm run server
   

2. Frontend:

   cd "PocketLM UI"
   npm run dev
   

Access the UI at http://localhost:3000. Everything Now by Arcade Fire