# Methodology

## LLM Model Comparison for Enterprise Use Cases (2026)

Published by [Salt Technologies AI](https://www.salttechno.ai) | Version: Q1 2026 v2 | Last updated: February 18, 2026

---

## Data Categories

This dataset combines three categories of independently sourced and verified data.

### 1. Specifications and Pricing

- Sourced directly from **official provider documentation** and API pricing pages as of February 2026
- Pricing reflects **pay-as-you-go API rates** in USD
- Volume discounts, committed-use pricing, and prompt caching discounts are excluded from the primary cost fields (noted in `pricingNote` where applicable)
- Open-source model pricing reflects **median costs across major inference providers**: Together AI, Groq, Fireworks AI, and DeepInfra
- Context window sizes, parameter counts, training cutoffs, and feature flags reflect documented specifications

### 2. Benchmark Scores

Four industry-standard benchmarks are included:

| Benchmark | What It Measures | Scale | Source |
|---|---|---|---|
| **MMLU** | Massive Multitask Language Understanding | 0-100 | Original papers, provider reports |
| **HumanEval** | Code generation accuracy | 0-100 | Original papers, provider reports |
| **MATH** | Mathematical problem solving | 0-100 | Original papers, provider reports |
| **MT-Bench** | Multi-turn instruction following | 0-10 | LMSYS Chatbot Arena |

**Score sourcing priority:**
1. Official provider-reported scores from technical reports
2. Verified third-party evaluations (LMSYS Chatbot Arena, Stanford HELM, Artificial Analysis)
3. Reproduced results from peer-reviewed evaluations

**Null values** indicate the provider has not published a verified score for that benchmark. We do not estimate or interpolate missing scores.

**Important note:** As the industry transitions to newer evaluation suites (MMLU Pro, SWE-bench, GPQA Diamond), traditional benchmark comparisons across model generations may reflect different evaluation conditions.

### 3. Latency and Throughput

- **Time-to-first-token (TTFT)** and **throughput (tokens per second)** are measured using standardized prompts:
  - Input: 500 tokens
  - Output: 200 tokens
- Measured against each provider's **production API endpoint from US-East regions**
- Results represent the **median of 100 sequential requests** during off-peak hours
- Self-hosted and inference-provider latency varies by hardware and provider; ranges reflect typical deployments on **H100/H200 GPUs**

### 4. API Feature Flags

Feature flags (`functionCalling`, `jsonMode`, `streaming`, `fineTuning`, `enterpriseReady`) reflect **documented GA (Generally Available) features** as of the dataset date. Beta or preview features are excluded.

## Models Included (16)

| Provider | Models | Count |
|---|---|---|
| OpenAI | GPT-4.1, GPT-4.1 mini, o3, o4-mini | 4 |
| Anthropic | Claude Sonnet 4.5, Claude Haiku 4.5, Claude Opus 4.5 | 3 |
| Google | Gemini 2.5 Pro, Gemini 2.5 Flash | 2 |
| Meta | Llama 4 Scout, Llama 4 Maverick | 2 |
| DeepSeek | DeepSeek V3, DeepSeek R1 | 2 |
| Mistral AI | Mistral Large 3, Mistral Small 3.2 | 2 |
| Cohere | Command A | 1 |

### Selection Criteria

Models are included if they meet at least two of:
- Available via production API with published pricing
- Widely used in enterprise deployments (based on Salt Technologies AI project data)
- Top-5 performance on at least one included benchmark
- Open-source with significant community adoption

### Excluded Models

- Research-only models without production APIs
- Models deprecated before the dataset date
- Provider-specific fine-tuned variants
- Models with fewer than 7B parameters (outside enterprise scope)

## Pricing Methodology

| Model Type | Pricing Basis |
|---|---|
| Commercial API (OpenAI, Anthropic, Google, Cohere) | Official API pricing page, pay-as-you-go tier |
| Open source via inference providers (Llama 4, DeepSeek, Mistral) | Median price across Together AI, Groq, Fireworks AI, DeepInfra |

Costs are per 1 million tokens. To estimate monthly costs:

```
Monthly cost = (conversations/month) × (avg tokens/conversation) × (cost per 1M tokens / 1,000,000)
```

**Example:** 10,000 conversations/month at 2,000 tokens each with GPT-4.1:
- Input: 10,000 × 1,000 × ($2.00 / 1,000,000) = $20.00
- Output: 10,000 × 1,000 × ($8.00 / 1,000,000) = $80.00
- Total: ~$100/month

## Limitations

- Benchmark scores may not reflect real-world production performance for specific use cases
- Pricing changes frequently; verify current rates on provider pricing pages
- Latency measurements are point-in-time and vary by region, load, and prompt characteristics
- Parameter counts for closed-source models are undisclosed and estimated ranges are not included
- This dataset covers text-focused capabilities; specialized multimodal benchmarks (image, video, audio) are not included
- Enterprise readiness assessment is based on documented features, not independent audit

## Update Frequency

Updated **quarterly** to reflect new model releases, pricing changes, and benchmark updates. Major mid-quarter model launches may trigger an interim update (indicated by version suffix, e.g., "Q1 2026 v2").

## Contact

For questions about methodology, data corrections, or partnership inquiries:

- **Website:** [www.salttechno.ai](https://www.salttechno.ai)
- **Dataset page:** [www.salttechno.ai/datasets/llm-model-comparison-2026](https://www.salttechno.ai/datasets/llm-model-comparison-2026/)
- **Contact:** [www.salttechno.ai/contact](https://www.salttechno.ai/contact/)