The Hidden Bottleneck in Production AI Isn't the Model — It's Context Management

Alex R

arybach

Follow

Everyone is comparing models.

• GPT-5.6
• Claude Opus
• DeepSeek V4
• GLM-5.2
• Million-token context windows
• Reasoning benchmarks

Model quality matters.

But after building production AI systems, I kept running into a completely different bottleneck.

It wasn't the model. It was every decision surrounding the model.

Every production AI request makes the same decisions

Before a model generates a single token, the system has already answered questions like:

• Should retrieval be used?
• Which retrieval strategy?
• Which knowledge sources?
• Which memory should be searched?
• Should context be compressed?
• Should another model verify the answer?
• Which model is appropriate?
• How much latency and cost is acceptable?

Most applications answer these questions with hardcoded logic.

if enterprise_customer:
    use_gpt5()

if support_query:
    top_k = 10

if code_question:
    rerank = True

The problem is that these decisions are not static. The optimal strategy depends on:

• user intent
• available knowledge
• latency constraints
• cost constraints
• historical performance
• previous execution outcomes

Static pipelines eventually become technical debt.

Every capability bolted onto the pipeline becomes another branch of application code — the maintenance burden, not the product.

Bigger context windows don't solve this

Long-context models are an impressive engineering achievement. They reduce many limitations of traditional RAG.

They do not answer a more fundamental question: what information should the model see?

More context does not automatically produce better answers. It often produces:

• higher latency
• higher inference cost
• more attention competition
• more irrelevant information

The optimization problem simply moves.

Retrieval isn't the bottleneck anymore

While benchmarking heterogeneous datasets (financial and medical records), one observation kept repeating.

The correct document was usually retrieved. The model still produced worse answers.

Why? Because it also received documents from unrelated domains. The answer wasn't missing — the context was polluted.

Routing retrieval by domain eliminated cross-domain noise while preserving recall.

The problem wasn't retrieval. It was deciding which retrieval strategy to use.

Memory has the same problem

Conversation history is not memory. Most assistants search one long transcript.

In practice, memory naturally separates into different types:

• recent conversation
• long-term semantic knowledge
• persistent entities

Different questions require different memories. Searching all of them every time increases latency and often introduces irrelevant context.

Memory should be routed, not simply stored.

The pattern repeats everywhere

The same optimization problem appears across production AI systems:

• retrieval
• memory
• model routing
• verification
• execution planning
• tool selection

Each subsystem makes decisions independently. Very few optimize them together.

The database analogy

Applications don't tell PostgreSQL which index to scan. They describe intent. The query planner determines the execution strategy.

I believe AI infrastructure is evolving toward the same abstraction. Applications should describe intent. Infrastructure should determine:

• which retrieval strategy
• which model
• which memory
• which verification policy
• which execution graph

The same abstraction relational databases have had since 1979 — you write intent; the planner decides execution.

Context Runtime

This observation led me to build Context Runtime.

Instead of another RAG framework, it provides a planning layer that evaluates multiple execution strategies before any model is called.

Each execution is measured. The runtime learns which strategies perform best for different request types and continuously improves future decisions.

Intent in, a cost-optimized execution graph out — verified before a single token is generated, and improved after every run.

The implementation is available in both Python and Go.

Closing thoughts

Model capabilities will continue improving. Context windows will continue growing. Inference will continue becoming cheaper.

None of those eliminate the need to decide: what should the model see?

I think context management will become one of the defining infrastructure problems of production AI.

References

• Whitepaper — https://redevops.io/whitepaper
• GitHub — https://github.com/redevops-io/context-runtime