πŸ›‘οΈ MyGuard AI Document Security Gateway - FastAPI ML Microservice

High-Performance RETVec + CNN Text Classification Microservice for Prompt Injection & Document Threat Defense

fastapi tensorflow retvec keras scikit-learn pydantic firebase-admin supabase uvicorn python-dotenv Docker Swagger Hugging Face GitHub

---

πŸ“Œ Executive Summary

MyGuard AI Document Security Gateway ML Service is a stateless, high-throughput Machine Learning microservice built with Python 3.10+, FastAPI, TensorFlow, and Google RETVec. It serves as the dedicated Layer 2 ML Classifier within the broader MyGuard AI Document Security infrastructure.

MyGuard FastAPI ML Service Swagger API Documentation

As enterprise organizations ingest unstructured documents (PDF, DOCX, PPTX, XLSX, TXT) into Large Language Model (LLM) agents and RAG (Retrieval-Augmented Generation) Knowledge Graphs, adversaries attempt to inject malicious payloads (Indirect Prompt Injections, Jailbreaks, System Override Attacks, and Data Exfiltration Commands).

This microservice analyzes extracted document text, optical OCR text streams, and steganographically hidden text layers, evaluating them through a character-level RETVec + Conv1D Deep Neural Network. It operates completely free of external LLM API calls, delivering zero-latency, deterministic threat classification before forwarding suspicious items for downstream LLM evaluation.

Model Readiness & Dataset Scaling Notice:

  • Architecture & Pipeline Readiness: The model architecture (Google RETVec + Conv1D dual-head neural network) is fully implemented, deployed, and ready for real-time threat inference.
  • Dataset Volume & Diversity Bottleneck: To further improve model accuracy, the primary requirement is expanding dataset volume and sample diversity. As training materials grow in both quantity and quality (incorporating diverse real-world documents and injection techniques), model performance will scale accordingly.
  • Private Service Architecture & Testing Mode: In a production environment, this ML microservice operates as a network-isolated Private Microservice protected by X-Internal-Token. For jury evaluation and live testing convenience via Swagger UI, evaluation endpoints have been temporarily made publicly accessible.

🌐 Project Ecosystem & Live Deployment Links

The MyGuard platform consists of synchronized web applications, core gateway backends, ML microservices, and file collection infrastructure:

πŸ”— Repositories, Live Platforms & Model Hubs

Component Name Type GitHub Repository & Model Hub Links
Python FastAPI ML Microservice & AI Model AI Model Backend & Weights GitHub Repository | πŸ€— Hugging Face Model Hub | Live Swagger
Node.js Gateway Backend Gateway REST API GitHub Repository | Live Swagger
MyGuard Web Frontend Web Application GitHub Repository | Live Portal

πŸš€ Production Live URLs & API Gateways

  • πŸ€— Hugging Face Model Hub (Model Card & Weights): https://huggingface.co/MegrurNiftiyev/MyGuard-Prompt-Injection-Detector
  • πŸ™ GitHub Repository (Source Code): https://github.com/MegrurNiftiyev/MyGuard-AI-Backend
  • 🐍 Python FastAPI ML Microservice (Production): https://myguard-ai-backend.onrender.com
  • πŸ“– ML Microservice Interactive Swagger UI Docs: https://myguard-ai-backend.onrender.com/api-docs
  • πŸš€ Node.js Gateway REST API Base URL (Production): https://mygurad-backend-v2.onrender.com/api
  • πŸ“– Node.js Gateway Interactive Swagger UI Docs: https://mygurad-backend-v2.onrender.com/api-docs
  • ⚑ Real-Time WebSocket Server (Socket.IO): https://mygurad-backend-v2.onrender.com

🧠 Deep-Dive Machine Learning (ML) Mechanism & Architecture

This microservice uses a specialized Dual-Output Deep Learning Model that combines Google's RETVec (Resilient Equivariant Text Vectorizer) with a 1D Convolutional Neural Network (CNN).

[ Raw Input Text Stream (PDF / OCR / Hidden Text) ]
                        β”‚
                        β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ RETVec Tokenizer (Sequence Length = 128)                     β”‚
β”‚  - Character-level & byte-level embedding graph              β”‚
β”‚  - Adversarial typo & visual obfuscation resistance           β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                        β”‚
                        β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ 1D Convolutional Layer (128 Filters, Kernel Size = 5, ReLU) β”‚
β”‚  - Spatial character-level n-gram feature extraction          β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                        β”‚
                        β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Global MaxPooling 1D                                         β”‚
β”‚  - Position-invariant maximum feature activation selection   β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                        β”‚
                        β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Dense Trunk (64 Units, ReLU) + Dropout (0.3 Rate)            β”‚
β”‚  - Shared non-linear feature representation                  β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
            β”‚                                      β”‚
            β–Ό                                      β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”            β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Head 1: Risk Label      β”‚            β”‚ Head 2: Attack Category β”‚
β”‚ Dense(3, Softmax)       β”‚            β”‚ Dense(6, Sigmoid)       β”‚
β”‚  - safe                 β”‚            β”‚  - Instruction Override β”‚
β”‚  - suspicious           β”‚            β”‚  - Ranking Manipulation β”‚
β”‚  - injection            β”‚            β”‚  - Data Exfiltration    β”‚
β”‚ Loss: Categorical Cross β”‚            β”‚  - Social Engineering   β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜            β”‚  - Prompt Leaking       β”‚
                                       β”‚  - Context Manipulation β”‚
                                       β”‚ Loss: Binary Cross      β”‚
                                       β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

πŸ–ΌοΈ Deep Learning Model Computational Graph & Architecture Diagram

MyGuard RETVec + 1D CNN Model Architecture

πŸ”¬ Detailed Layer-by-Layer Architectural Specification

Layer Name Output Tensor Shape Config & Activation Purpose & Security Role
text_input (batch_size, 1) UTF-8 String Input Accepts raw text generated by 60-word sliding window chunker
RETVecTokenizer (batch_size, 128, 256) seq_len=128, 256-dim Google RETVec character/byte embedding resilient to typos/obfuscation
Conv1D (batch_size, 124, 128) 128 filters, kernel=5, ReLU Extracts spatial 5-gram character sequence patterns of prompt overrides
GlobalMaxPooling1D (batch_size, 128) Channels-last Max Pool Position-invariant downsampling capturing peak threat activations
Dense Trunk (batch_size, 64) 64 units, ReLU, Dropout=0.3 Non-linear feature fusion & regularization layer preventing overfitting
categories Head (batch_size, 6) 6 units, Sigmoid Multi-label attack taxonomy head classifying 6 threat categories
label Head (batch_size, 3) 3 units, Softmax Primary risk severity classification head (safe, suspicious, injection)

1. Google RETVec Tokenization (Character-Level Embeddings)

Traditional NLP vectorizers (Word2Vec, GloVe, BERT) rely on token vocabularies. Adversaries exploit this vulnerability by injecting zero-width spaces, leetspeak (p r 0 m p t i n j 3 c t 1 o n), homoglyphs, or steganographic unicode modifications that cause subword tokenizers to split words into benign sub-tokens.

RETVec (Resilient Equivariant Text Vectorizer) solves this by embedding text directly at the byte and character level inside the TensorFlow graph:

  • Sequence Length: 128 character tokens per chunk.
  • Robustness: Equivariant architecture produces consistent numeric vector representations even when characters are swapped, substituted, or obfuscated.
  • Embedded Graph: RETVec is compiled directly into the SavedModel, eliminating external preprocessing dependencies during production inference.

2. 1D Convolutional Neural Network (CNN) Trunk

The embedded vector sequence passes through a lightweight, high-speed 1D CNN:

  • Conv1D(128, kernel_size=5, activation='relu'): Captures spatial 5-gram character sequence patterns associated with command injection syntax ("ignore previous instructions", "system prompt override", "print secret key").
  • GlobalMaxPooling1D(): Downsamples feature maps by extracting the maximum activation score, making threat detection invariant to the offset or positioning of the injection within a text segment.
  • Dense(64, activation='relu') & Dropout(0.3): Dense representation layer with 30% dropout regularization to prevent overfitting on specific phrasing.

3. Dual Classification Output Heads

The network splits into two independent heads to serve different risk management operations:

Head 1: Risk Severity Label (label)

  • Activation: 3-class Softmax
  • Output Classes:
    • safe: Benign, standard business text.
    • suspicious: Ambiguous or subtle text requiring escalation.
    • injection: High-confidence prompt override or malicious attack payload.
  • Loss Function: categorical_crossentropy

Head 2: Multi-Label Attack Taxonomy (categories)

  • Activation: 6-unit Sigmoid (Multi-label classification, threshold = 0.5)
  • Output Categories:
    1. Instruction Override: Overriding system prompt rules.
    2. Ranking Manipulation: Distorting AI scoring or review outcomes.
    3. Data Exfiltration: System prompt leaking or credentials theft.
    4. Social Engineering: Phishing, coercion, or pretexting prompts.
    5. Prompt Leaking: Direct attempts to expose backend instructions.
    6. Context Manipulation: Injecting false context into LLM memory frames.
  • Loss Function: binary_crossentropy

4. Zero-Trust Security Posture & Loss Functions

In enterprise security gateways, a False Negative (missing a malicious injection) is a critical vulnerability, whereas a False Positive (flagging a safe document as suspicious) simply routes the file to Layer 3 (LLM Review) for confirmation.

  • Class Weighting: Uses sklearn.utils.class_weight.compute_class_weight during training to assign higher loss penalization to missed injection samples.
  • Recall Optimization: The network thresholding is tuned specifically for 100% Injection Recall, ensuring zero malicious payloads bypass Layer 2 undetected.

πŸ“Š Dataset Processing, Extraction Pipeline & Real Evaluation

1. Document Extraction & Multi-Format Ingestion

The dataset pipeline (app/scripts/train_model.py and app/services/supabase_dataset.py) handles structured parsing across large-scale synthetic datasets and real-world administrative files:

  • 10,200 PDF Synthetic Injection Dataset v4: 10,200 synthetic PDF documents generated across 6 document archetypes (invoice, contract, report, email, resume, form) with 1,700 clean baselines and 8,500 prompt injection attacks (invisible_text, system_spoof, goal_hijacking, persona_swap, metadata).
  • Real Azerbaijani & English Administrative Documents: 325 real-world government and corporate documents (Baku IH, Ministries, Town Councils, Expense Reports).
  • Microsoft Word (.docx): Parsed paragraph-by-paragraph and cell-by-cell across nested tables (python-docx).
  • PowerPoint (.pptx): Text frames and speaker notes extracted across slides (python-pptx).
  • Adobe PDF (.pdf): Structural text stream and binary metadata extraction (pypdf).
  • Archive Packages (.zip): Recursive decompression and text stream extraction.
  • Plain Text (.txt): UTF-8 stream normalization.

2. Sliding-Window Text Chunking Algorithm

Prompt injections are often hidden deep within long, multi-page corporate documents. Feeding an entire 50-page document as one block dilutes the injection signal.

The training and inference engine implements a sliding-window text chunker:

  • Chunk Size: 60 words
  • Overlap Size: 30 words
  • Mechanism: Text is segmented into overlapping windows. If any single chunk triggers an injection classification above the threshold, the document is flagged as injection.
def chunk_text(text: str, chunk_size: int = 60, overlap: int = 30) -> list[str]:
    lines = [line.strip() for line in text.split("\n") if line.strip()]
    chunks = []
    for line in lines:
        words = line.split()
        if len(words) <= chunk_size:
            chunks.append(line)
        else:
            i = 0
            while i < len(words):
                c = " ".join(words[i:i + chunk_size])
                chunks.append(c)
                i += chunk_size - overlap
    return chunks

3. Supabase Cloud Data Synchronization

Dataset files are maintained in Supabase Cloud Storage and Firestore/PostgreSQL tables. Calling POST /api/v1/dataset/sync downloads missing samples into local storage (./data/raw/benign and ./data/raw/injection).


πŸ“ˆ Real Dataset Evaluation Report & Benchmark Metrics

  • Training Chunks Total: 1,816 chunks (1,072 safe, 744 injection).
  • Held-Out Test Set: 6 real-world complete document files (3 clean Azerbaijani/English documents, 3 malicious injection documents) kept completely isolated from training.

Held-Out Test Evaluation Results (2026-09-01 Run):

  • Total Test Documents: 6
  • Injection Detection Rate (Recall): 100.00% (3 out of 3 malicious injection files caught)
  • False Negative Rate: 0.00% (Zero missed threats)
  • Model Posture: Strict Security Mode (Zero-Trust)

Per-File Inference Breakdown Table:

File Name Expected Predicted Label Evaluation Status Safe Prob Suspicious Prob Injection Prob Max Chunk Inj Prob
09_Official_Letter_Clean.docx safe injection Strict Flag (FP) 84.04% 0.00% 15.96% 52.29%
10_Meeting_Minutes_Clean.docx safe injection Strict Flag (FP) 83.99% 0.00% 16.01% 51.19%
Monthly_Financial_Expense_Report.pdf safe injection Strict Flag (FP) 90.64% 0.00% 9.36% 62.52%
01_Monthly_Activity_Report_Injection.docx injection injection βœ“ PASSED 75.20% 0.00% 24.80% 92.98%
16_Travel_Expenses_Stealth_Injection.docx injection injection βœ“ PASSED 69.57% 0.00% 30.43% 72.35%
19_Purchase_Order_Injection.docx injection injection βœ“ PASSED 78.84% 0.00% 21.16% 78.69%

⚑ 3-Layer Hybrid Security Pipeline Integration

The FastAPI ML service operates seamlessly inside the 3-Layer MyGuard Security Architecture:

[ Document Upload via Node.js Gateway ]
                  β”‚
                  β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ LAYER 1: Heuristic & Visual Diff Detection (Node.js)         β”‚
β”‚  - Raw PDF Text Layer vs. Optical Tesseract OCR Text         β”‚
β”‚  - Zero-opacity font & white-on-white steganography scan      β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                        β”‚
                        β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ LAYER 2: RETVec+CNN ML Microservice (Python FastAPI)         β”‚
β”‚  - Fast character-level Deep Learning classification         β”‚
β”‚  - Dual-head risk scoring & attack vector categorization     β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                        β”‚
                        β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
                        β”‚ (Result = safe)          β”‚ (Result = suspicious / injection)
                        β–Ό                          β–Ό
            [ ALLOW / PROCEED ]      β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
                                     β”‚ LAYER 3: LLM Review      β”‚
                                     β”‚ (OpenAI gpt-4o-mini)     β”‚
                                     β”‚ Deep semantic evaluation β”‚
                                     β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                                                   β”‚
                                                   β–Ό
                                         [ SANITIZE / BLOCK ]

πŸ—„οΈ Model Registry & Persistence Architecture

To guarantee resiliency, full model auditability, and fast container startup on platforms like Render:

  1. Local Model Directory (data/models/): All historical model version files (model_run-01.keras through model_run-11.keras) are saved and version-tagged locally under ./data/models/. Whenever a new training run completes, it automatically saves a new versioned file (e.g., model_run-12.keras).
  2. Active Model File & Cache:
    • data/cache/active_model.keras: Represents the currently active model loaded into memory for real-time /analyze-injection inference (0 ms load).
    • data/models/retvec_cnn_model.keras: Serves as the primary active local Keras model artifact.
  3. Firebase Storage Persistence: Trained models are archived as ZIP files (models/model_<version>.zip) and uploaded to Firebase Storage.
  4. Firebase Firestore Registry: Active, candidate, and archived model versions are registered in the models Firestore collection:
    interface ModelMetadata {
      version: string;             // e.g., "run-11"
      status: 'active' | 'candidate' | 'archived';
      isCurrentVersion: boolean;   // true for the active model
      sourceCommit?: string;       // Git commit hash (e.g., "42743dc")
      description?: string;        // Detailed dataset & test metrics summary
      storagePath: string;         // Firebase Storage path
      metrics: {
        test_acc: number;
        recall: number;
        train_loss: number;
      };
      createdAt: string;
    }
    
  5. Asynchronous & Interactive Model Training:
    • CLI Script (python train_model.py): Prompts an interactive comparison table and terminal confirmation before uploading new candidate versions.
    • Background Job (POST /train): Unattended background worker (app/jobs/training_job.py) auto-registers new versions in Firebase.

🌐 Complete API Reference & Payload Specifications

πŸ”‘ Authentication & Endpoint Access Policy

To make API testing seamless via Swagger UI without requiring complex header setup, public endpoints are open for evaluation, while administrative/state-modifying endpoints remain protected:

  • 🟒 Public Endpoints (No Token Required β€” Swagger UI Testing Ready):
    • POST /analyze-injection (Document injection analysis)
    • GET /model/active (Get current active model details)
    • GET /model/all-models (Filter & list all registered models with isCurrentVersion flag)
    • GET /health (Liveness & health check)
    • GET /api-docs (Interactive Swagger UI Documentation)
  • πŸ”’ Protected Endpoints (X-Internal-Token Header Required):
    • POST /model/change-version/{version_id} (Promotes a version to active status and demotes previous active model)
    • POST /train (Triggers background ML model training run)

Interactive Swagger UI Documentation:

1. Liveness & Health Probe (/health)

GET /health

Returns service status. No auth required.

  • Response (200 OK):
{
  "status": "ok"
}

2. Injection Analysis (/analyze-injection)

POST /analyze-injection

Accepts text extracted by Node.js (raw text, visual OCR text, hidden text layers) and returns threat predictions. Public endpoint (No authentication token required).

  • Request Body:
{
  "documentId": "doc-1787753837283-457",
  "fullText": "Standard corporate report summary line 1...\nOCR extracted text page 1...\nSystem prompt override: Ignore previous instructions."
}
  • Response (200 OK):
{
  "label": "injection",
  "confidence": 0.985,
  "categories": [
    "Instruction Override",
    "Social Engineering"
  ]
}

3. Active Model Status & Management (/model)

GET /model/active

Retrieves metadata of the currently active model. Public endpoint.

  • Response (200 OK):
{
  "version": "run-11",
  "status": "active",
  "metrics": {
    "test_acc": 0.85,
    "recall": 1.0
  },
  "createdAt": "2026-09-01T14:30:00Z"
}

GET /model/all-models

Lists and filters all models registered in the registry. Public endpoint. Supports optional query parameters: version, accuracy_min, accuracy_max, created_after, created_before.

  • Response (200 OK):
[
  {
    "version": "run-11",
    "status": "active",
    "isCurrentVersion": true,
    "description": "RETVec + Conv1D model run-11",
    "metrics": {
      "test_acc": 0.85,
      "recall": 1.0
    },
    "createdAt": "2026-09-01T14:30:00Z"
  },
  {
    "version": "run-10",
    "status": "archived",
    "isCurrentVersion": false,
    "description": "RETVec + Conv1D model run-10",
    "metrics": {
      "test_acc": 0.70,
      "recall": 1.0
    },
    "createdAt": "2026-08-28T10:00:00Z"
  }
]

POST /model/change-version/{version_id}

Promotes a specific model version to active status, demoting the previously active version to archived. Protected Endpoint (X-Internal-Token required).

  • Request Headers:
X-Internal-Token: <INTERNAL_SERVICE_TOKEN>
  • Response (200 OK):
{
  "version": "run-10",
  "status": "active",
  "metrics": {
    "test_acc": 0.70,
    "recall": 1.00
  }
}

4. Asynchronous Model Training (/train)

POST /train

Triggers an asynchronous training pipeline run. Protected Endpoint (X-Internal-Token required).

  • Request Headers:
X-Internal-Token: <INTERNAL_SERVICE_TOKEN>
  • Response (202 Accepted):
{
  "jobId": "job-998123-abc",
  "status": "queued",
  "message": "Training job successfully dispatched to background runner."
}

5. Supabase Dataset Management (/api/v1/dataset)

GET /api/v1/dataset/files

Lists clean (benign) and malicious (injection) dataset files in Supabase.

POST /api/v1/dataset/sync

Synchronizes remote Supabase dataset files to local disk.

  • Response (200 OK):
{
  "status": "success",
  "message": "Dataset successfully synchronized from Supabase.",
  "synced_counts": {
    "benign": 1072,
    "injection": 744
  }
}

πŸ›‘οΈ Security & Authentication Architecture

To prevent unauthorized access and Denial-of-Service (DoS) abuse:

  1. Private Microservice Isolation Mode:
    • In production deployment environments, this ML microservice is deployed as an internal Private Service accessible only within the internal virtual network (VPC).
    • In live evaluation mode, public access is temporarily enabled for evaluation endpoints to allow zero-friction testing via Swagger UI.
  2. Header Authentication: Protected endpoints validate the X-Internal-Token header against INTERNAL_SERVICE_TOKEN for server-to-server commands (POST /train, POST /model/change-version/{version_id}).
  3. Automated IP Ban Enforcement:
    • Tracks failed authentication attempts per client IP in memory (app/api/dependencies.py).
    • If an IP exceeds 3 invalid token attempts, it is added to the banned IP registry.
    • Subsequent requests from banned IPs return HTTP 403 Forbidden instantly.

🧱 Complete Project Structure

Ai-Models
β”œβ”€β”€ .env.example                # Template environment configuration
β”œβ”€β”€ .gitignore                  # Git exclude rules
β”œβ”€β”€ Dockerfile                  # Containerization directives
β”œβ”€β”€ NODE_JS_INTEGRATION_GUIDE.md # Node.js gateway integration manual
β”œβ”€β”€ README.md                   # Primary documentation
β”œβ”€β”€ REAL_DATASET_TRAINING_REPORT.md # Training report & metric log
β”œβ”€β”€ requirements.txt            # Python package dependencies
β”œβ”€β”€ train_model.py              # CLI entrypoint wrapper (delegates to app.scripts.train_model)
β”œβ”€β”€ seed_model.py               # CLI entrypoint wrapper (delegates to app.scripts.seed_model)
β”œβ”€β”€ push_to_firebase.py         # CLI entrypoint wrapper (delegates to app.scripts.push_to_firebase)
β”œβ”€β”€ app/
β”‚   β”œβ”€β”€ main.py                 # FastAPI application factory & lifecycle hooks
β”‚   β”œβ”€β”€ api/
β”‚   β”‚   β”œβ”€β”€ dependencies.py     # Auth verification & IP ban protection
β”‚   β”‚   └── routes/
β”‚   β”‚       β”œβ”€β”€ classify.py     # POST /analyze-injection route handler
β”‚   β”‚       β”œβ”€β”€ model_status.py # GET/PATCH /model endpoints
β”‚   β”‚       └── train.py        # POST /train background runner route
β”‚   β”œβ”€β”€ core/
β”‚   β”‚   β”œβ”€β”€ config.py           # Pydantic Settings & Env configuration
β”‚   β”‚   β”œβ”€β”€ firebase.py         # Firebase Admin SDK initialization
β”‚   β”‚   └── logging.py          # Structured JSON logging setup
β”‚   β”œβ”€β”€ jobs/
β”‚   β”‚   └── training_job.py     # Background worker thread for training runs
β”‚   β”œβ”€β”€ ml/
β”‚   β”‚   β”œβ”€β”€ cnn/
β”‚   β”‚   β”‚   β”œβ”€β”€ architecture.py # RETVec + Conv1D model graph
β”‚   β”‚   β”‚   └── model_registry.py # Firebase & local disk load/save logic
β”‚   β”‚   β”œβ”€β”€ preprocessing/
β”‚   β”‚   β”‚   └── normalize.py    # Basic text normalization helpers
β”‚   β”‚   β”œβ”€β”€ retvec/
β”‚   β”‚   β”‚   └── tokenizer.py    # Google RETVec integration wrappers
β”‚   β”‚   └── training/
β”‚   β”‚       β”œβ”€β”€ dataset.py      # Stratified dataset split & loader
β”‚   β”‚       β”œβ”€β”€ evaluate.py     # Precision/Recall/F1 metrics computation
β”‚   β”‚       └── train.py        # Class weight computation & training loop
β”‚   β”œβ”€β”€ models/
β”‚   β”‚   └── schemas.py          # Pydantic request/response schemas
β”‚   β”œβ”€β”€ scripts/                # Standalone CLI scripts module
β”‚   β”‚   β”œβ”€β”€ push_to_firebase.py # Firebase model upload & promotion module
β”‚   β”‚   β”œβ”€β”€ seed_model.py       # Initial model seeding module
β”‚   β”‚   └── train_model.py      # RETVec+CNN training & held-out test pipeline
β”‚   └── services/
β”‚       └── supabase_dataset.py # Supabase Storage & DB dataset manager
β”œβ”€β”€ data/
β”‚   β”œβ”€β”€ cache/                  # Local model cache directory
β”‚   └── raw/                    # Local training dataset (benign/injection)
└── tests/                      # Pytest automated test suite
    β”œβ”€β”€ test_classify.py
    β”œβ”€β”€ test_model_registry.py
    └── test_training.py

βš™οΈ Environment Variables Reference

Create a .env file in the project root based on .env.example:

# Shared Secret for Service-to-Service Authorization
INTERNAL_SERVICE_TOKEN=myguard-internal-secret-token-2026

# Server Bind Settings
PORT=8000
HOST=0.0.0.0
LOG_LEVEL=INFO

# Firebase Admin SDK Credentials & Storage Bucket
FIREBASE_CREDENTIALS_PATH=./mygurad-firebase-admin.json
FIREBASE_STORAGE_BUCKET=myguard-app.appspot.com

# Supabase Data Pipeline Credentials
SUPABASE_URL=https://your-supabase-project.supabase.co
SUPABASE_SERVICE_ROLE_KEY=your-supabase-service-role-key
SUPABASE_STORAGE_BUCKET=team-files

# CORS Allowed Origins
ALLOWED_ORIGINS=https://mygurad-backend-v2.onrender.com,http://localhost:8000

πŸ’» Setup, Installation & Execution

1. Clone Repository

git clone https://github.com/MegrurNiftiyev/MyGuard-AI-Backend.git
cd IDDA-Final-Project-Ai-Backend

2. Set Up Virtual Environment & Dependencies

python -m venv venv
# On Windows:
venv\Scripts\activate
# On Linux/macOS:
source venv/bin/activate

pip install -r requirements.txt

3. Environment Configuration

cp .env.example .env

4. Bootstrap Model (Optional for local testing)

python seed_model.py

5. Run FastAPI Application locally

uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload

Interactive Swagger UI will be available at: http://localhost:8000/api-docs

6. Train Model on Dataset

python train_model.py

7. Run Container with Docker

docker build -t myguard-ai-backend .
docker run -p 8000:8000 --env-file .env myguard-ai-backend

πŸ›‘οΈ Error Handling Architecture

All API error responses follow a standardized JSON structure:

{
  "detail": {
    "error": "Short description of failure",
    "detail": "Detailed message"
  }
}
HTTP Status Category Failure Condition
401 Unauthorized Missing or invalid X-Internal-Token header
403 Forbidden Client IP banned after 3 failed auth attempts
404 Not Found Requested dataset record or model version not found
500 Internal Error Internal server or training job failure
503 Unavailable Classification model not initialized or unavailable

🐳 Docker Containerization & Production Deployment

The microservice includes a lightweight, multi-stage Dockerfile for enterprise containerization and zero-dependency cloud deployments (Render, AWS ECS, GCP Cloud Run, Kubernetes):

1. Build Docker Image

docker build -t myguard-ai-backend .

2. Run Container Locally

docker run -d -p 8000:8000 --env-file .env --name myguard-ai-backend myguard-ai-backend

3. Verify Container Health

curl http://localhost:8000/health

πŸ“œ License

Licensed under the MIT License.

Downloads last month
299
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Evaluation results

  • recall on MyGuard Real Administrative Document Dataset & PDF Synthetic Dataset v4
    self-reported
    1.000
  • accuracy on MyGuard Real Administrative Document Dataset & PDF Synthetic Dataset v4
    self-reported
    0.850