MEMTRACK: Evaluating Long-Term Memory and State Tracking in
  Multi-Platform Dynamic Agent Environments
Authors: name: Darshan Deshpande; name: Varun Gangal; name: Hersh Mehta; name: Anand Kannappan; name: Rebecca Qian; name: Peng Wang
Published: 2025-10-01 18:34:03

Abstract: Recent works on context and memory benchmarking have primarily focused on
conversational instances but the need for evaluating memory in dynamic
enterprise environments is crucial for its effective application. We introduce
MEMTRACK, a benchmark designed to evaluate long-term memory and state tracking
in multi-platform agent environments. MEMTRACK models realistic organizational
workflows by integrating asynchronous events across multiple communication and
productivity platforms such as Slack, Linear and Git. Each benchmark instance
provides a chronologically platform-interleaved timeline, with noisy,
conflicting, cross-referring information as well as potential
codebase/file-system comprehension and exploration. Consequently, our benchmark
tests memory capabilities such as acquistion, selection and conflict
resolution. We curate the MEMTRACK dataset through both manual expert driven
design and scalable agent based synthesis, generating ecologically valid
scenarios grounded in real world software development processes. We introduce
pertinent metrics for Correctness, Efficiency, and Redundancy that capture the
effectiveness of memory mechanisms beyond simple QA performance. Experiments
across SoTA LLMs and memory backends reveal challenges in utilizing memory
across long horizons, handling cross-platform dependencies, and resolving
contradictions. Notably, the best performing GPT-5 model only achieves a 60\%
Correctness score on MEMTRACK. This work provides an extensible framework for
advancing evaluation research for memory-augmented agents, beyond existing
focus on conversational setups, and sets the stage for multi-agent,
multi-platform memory benchmarking in complex organizational settings