Context Is What You Need: The Maximum Effective Context Window for Real
  World Limits of LLMs
Authors: name: Norman Paulsen

Large language model (LLM) providers boast big numbers for maximum context
window sizes. To test the real world use of context windows, we 1) define a
concept of maximum effective context window, 2) formulate a testing method of a
context window's effectiveness over various sizes and problem types, and 3)
create a standardized way to compare model efficacy for increasingly larger
context window sizes to find the point of failure. We collected hundreds of
thousands of data points across several models and found significant
differences between reported Maximum Context Window (MCW) size and Maximum
Effective Context Window (MECW) size. Our findings show that the MECW is, not
only, drastically different from the MCW but also shifts based on the problem
type. A few top of the line models in our test group failed with as little as
100 tokens in context; most had severe degradation in accuracy by 1000 tokens
in context. All models fell far short of their Maximum Context Window by as
much as 99 percent. Our data reveals the Maximum Effective Context Window
shifts based on the type of problem provided, offering clear and actionable
insights into how to improve model accuracy and decrease model hallucination
rates.