JBrightmanAI/GEmbedder

global-embedder

Multilingual Dense Text Embedding Family – Decoder‑Only, Instruction‑Aware, State‑of‑the‑Art

---

Model Overview

global-embedder is a family of multilingual text embedding models built on a decoder‑only Transformer architecture. Unlike traditional encoder‑based embedders, this model uses last‑token pooling and L2 normalization to produce dense, fixed‑size vector representations from long input sequences (up to 32,768 tokens). It is designed to serve as a universal backbone for retrieval, semantic similarity, clustering, classification, bitext mining, and reranking across over 80 languages.

The key differentiator lies in its instruction‑aware training: query‑side natural language instructions allow the model to adapt to diverse downstream tasks without fine‑tuning. By leveraging contrastive learning on a large‑scale mixture of multilingual datasets and knowledge distillation for the smaller variants, global-embedder achieves top performance on the Multilingual MTEB v2 benchmark as of its release date.

Variant	Parameters	Embedding Dim	Max Tokens	MTEB v2 Score
global-embedder-270m	270M	640	32,768	66.5
global-embedder-0.6b	0.6B	1,024	32,768	69.0
global-embedder-27b	27B	5,376	32,768	74.3

---

Intended Uses & Limitations

Primary Use Cases

• Information Retrieval – Dense passage retrieval for web search, FAQ matching, and enterprise knowledge bases.
• Semantic Similarity – Computing pairwise sentence/document similarity for deduplication or clustering.
• Classification & Clustering – Producing input features for downstream classifiers or unsupervised grouping.
• Bitext Mining – Aligning sentences between different languages for translation or parallel corpus creation.
• Reranking – Improving first‑stage retrieval results by re‑scoring candidates.

Out‑of‑Scope Scenarios

• Generative tasks (text summarisation, dialogue, translation) – this model is not designed for autoregressive generation.
• Real‑time latency‑sensitive applications without hardware acceleration – the 27B variant requires substantial compute.
• Tasks requiring explicit reasoning or fact‑grounded knowledge – embeddings capture semantic similarity, not factual correctness.

Known Biases and Limitations

• The training data, while multilingual, may contain regional and cultural biases that can surface in the embeddings.
• Performance varies across languages; high‑resource languages (English, Chinese, Spanish) generally achieve better scores than low‑resource ones.
• The model does not inherently distinguish between factual and hypothetical statements – it treats all input text semantically.

---

How to Use

Option 1: Sentence Transformers (Recommended)

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("global-embedder-27b", model_kwargs={"dtype": "auto"})

# Use a pre‑configured instruction for web search queries
queries = [
    "how much protein should a female eat",
    "summit define"
]
documents = [
    "As a general guideline, the CDC's average requirement of protein for women ages 19 to 70 is 46 grams per day...",
    "Definition of summit for English Language Learners: the highest point of a mountain..."
]

query_emb = model.encode(queries, prompt_name="web_search_query")
doc_emb = model.encode(documents)

scores = (query_emb @ doc_emb.T) * 100
print(scores.tolist())

For a custom instruction, pass the prompt parameter:

query_emb = model.encode(
    queries,
    prompt="Instruct: Retrieve semantically similar passages\nQuery: "
)

Option 2: Transformers (Raw Pooling)

import torch
import torch.nn.functional as F
from transformers import AutoTokenizer, AutoModel

def last_token_pool(last_hidden_states, attention_mask):
    left_padding = (attention_mask[:, -1].sum() == attention_mask.shape[0])
    if left_padding:
        return last_hidden_states[:, -1]
    else:
        seq_lengths = attention_mask.sum(dim=1) - 1
        batch_size = last_hidden_states.shape[0]
        return last_hidden_states[torch.arange(batch_size, device=last_hidden_states.device), seq_lengths]

def instruct_query(task, query):
    return f'Instruct: {task}\nQuery: {query}'

task = "Given a web search query, retrieve relevant passages"
queries = [instruct_query(task, q) for q in ["how much protein should a female eat", "summit define"]]
documents = ["...", "..."]

tokenizer = AutoTokenizer.from_pretrained("global-embedder-27b")
model = AutoModel.from_pretrained("global-embedder-27b", dtype="auto").cuda()
model.eval()

batch = tokenizer(queries + documents, max_length=32768, padding=True, truncation=True, return_tensors="pt")
batch = {k: v.cuda() for k, v in batch.items()}

with torch.no_grad():
    outputs = model(**batch)
    embeddings = last_token_pool(outputs.last_hidden_state, batch["attention_mask"])
    embeddings = F.normalize(embeddings, p=2, dim=1)

scores = (embeddings[:2] @ embeddings[2:].T) * 100
print(scores.tolist())

Inference Widget (Feature Extraction)

You can test the model directly on the Hub using the inference widget (if hosted). Select the feature-extraction pipeline and input your text. Note that the widget returns raw hidden states – for a pooled embedding, please use the code above.

---

Training Data

The model was trained on a large‑scale proprietary mixture of multilingual datasets encompassing:

• Web‑crawled corpora (general domain)
• Parallel bitext for cross‑lingual alignment
• Question‑answer pairs and search logs
• Public benchmark datasets (e.g., MS‑MARCO, Natural Questions, XQuAD, and others – anonymised)

All data underwent deduplication, language identification, and filtering for quality (e.g., removal of toxic or low‑perplexity content). The final training set covers more than 80 languages, with a balanced sampling strategy to mitigate the dominance of high‑resource languages.

---

Training Procedure

• Architecture: Decoder‑only Transformer with causal attention, adapted for embedding via last‑token pooling.
• Objective: Contrastive learning with in‑batch negatives and hard‑negative mining. Instruction‑based query formatting was applied during training.
• Optimizer: AdamW with weight decay ([WEIGHT_DECAY]) and linear warmup.
• Hyperparameters (varies by size – representative for 27B):
  • Batch size: [BATCH_SIZE] (global with gradient accumulation)
  • Learning rate: [LEARNING_RATE]
  • Epochs: [EPOCHS]
  • Precision: mixed‑precision (BF16)
• Hardware: Distributed training across [NUMBER_OF_GPUS] NVIDIA A100 (80GB) GPUs.
• Knowledge Distillation (for 270M and 0.6B variants): Teacher embeddings from the larger 27B model were used to augment the contrastive loss.

---

Evaluation Results

Performance measured on the Multilingual MTEB v2 benchmark, which covers retrieval, clustering, classification, similarity, and reranking tasks across 50+ languages. The reported score is the average across all tasks (macro‑average).

Variant	Average MTEB v2 Score
global-embedder-270m	66.5
global-embedder-0.6b	69.0
global-embedder-27b	74.3

Detailed per‑task breakdowns are available in the evaluation suite ([EVALUATION_SUITE_LINK]). Reproduced scores may vary by ±0.2 due to software versions.

---

Environmental Impact

Training the 27B variant required approximately [TRAINING_HOURS] hours on [NUMBER_OF_GPUS] A100 GPUs. Estimated CO₂ emissions (using the Machine Learning Impact calculator) are [CO2_EMISSIONS] kg CO₂ equivalent. We are committed to reducing future footprints via model distillation and efficient scaling.

---

Bias, Risks, and Limitations

• Geographic and Cultural Bias: The training data over‑represents Western and East Asian internet content, which may skew embeddings for underrepresented regions.
• Stereotypical Associations: As with all large language models, the embeddings may encode societal stereotypes present in the training corpora.
• Misuse Potential: The model could be used to amplify surveillance or profiling if applied to sensitive personal data without proper safeguards.
• Adversarial Robustness: The model has not been extensively tested against adversarial perturbations; embeddings might be vulnerable to deliberate input manipulation.

Mitigations: We recommend deploying the model with a bias‑audit layer for high‑stakes applications, and always combining with human‑in‑the‑loop oversight. Users should not rely solely on model outputs for critical decisions.