Youme21/mobileclip-s0-multilingual-text-tflite

MobileCLIP-S0 Multilingual Text Encoder (Int8 TFLite)

This repository contains an edge-optimized multilingual text encoder in TFLite (int8) aligned to Apple's MobileCLIP-S0 image space.

It maps queries in English, Arabic, French, Russian, and Spanish into the same 512-dimensional vector space as the original MobileCLIP-S0 image encoder. This enables multilingual cross-modal retrieval directly on mobile/edge devices without re-indexing existing image embeddings.

This model was built to power on-device photo search in Memoria.

---

📦 Files Included

• text_encoder_int8w.tflite — Int8 weight-quantized text encoder with baked embedding layer (~30–40 MB).
• spm.model — 48,000-token SentencePiece tokenizer (BPE, byte fallback).

---

⚙️ Model & Tokenizer Specifications

• Embedding Dimension: 512 floats (L2-normalized)
• Context Length: 77 tokens
• Tokenizer: SentencePiece (spm.model) with pad=0, unk=1, BOS=48000, EOS=48001
• Sequence Format: [BOS] + token_ids[:75] + [EOS] + [PAD... to 77]
• Inputs to TFLite:
  • token_ids: int32[1, 77]
  • eos_pos: int32[1] (index of the EOS token)

---

🚀 Quickstart: Python Inference (TFLite)

Install requirements:

pip install tensorflow sentencepiece numpy

Run inference:

import numpy as np
import sentencepiece as spm
import tensorflow as tf

MODEL_PATH = "text_encoder_int8w.tflite"
SPM_PATH = "spm.model"
CONTEXT_LENGTH = 77

# 1. Load Tokenizer
sp = spm.SentencePieceProcessor(model_file=SPM_PATH)
V = sp.get_piece_size()
BOS_ID, EOS_ID = V, V + 1

# 2. Load TFLite Model
interpreter = tf.lite.Interpreter(model_path=MODEL_PATH)
interpreter.allocate_tensors()
input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()

input_idx_ids = next(d["index"] for d in input_details if len(d["shape"]) == 2)
input_idx_eos = next(d["index"] for d in input_details if len(d["shape"]) == 1)

def encode_query(text: str) -> np.ndarray:
    sp_ids = sp.encode(text, out_type=int)[: CONTEXT_LENGTH - 2]
    
    # Construct sequence: [BOS] + tokens + [EOS]
    token_array = np.zeros((1, CONTEXT_LENGTH), dtype=np.int32)
    token_array[0, 0] = BOS_ID
    for i, tid in enumerate(sp_ids):
        token_array[0, i + 1] = tid

    eos_pos = len(sp_ids) + 1
    token_array[0, eos_pos] = EOS_ID
    eos_pos_array = np.array([eos_pos], dtype=np.int32)

    # Invoke interpreter
    interpreter.set_tensor(input_idx_ids, token_array)
    interpreter.set_tensor(input_idx_eos, eos_pos_array)
    interpreter.invoke()

    emb = interpreter.get_tensor(output_details[0]["index"])[0]
    norm = np.linalg.norm(emb)
    return emb / norm if norm > 0 else emb

# Test cross-lingual similarity
q_en = encode_query("a cat sleeping on a sofa")
q_ar = encode_query("قطة نائمة على أريكة")
q_fr = encode_query("un chat dormant sur un canapé")
q_ru = encode_query("кошка спит на диване")
q_es = encode_query("un gato durmiendo en un sofá")

print("Arabic vs English Cosine:", np.dot(q_en, q_ar))
print("French vs English Cosine:", np.dot(q_en, q_fr))