From Tools to Teammates: Evaluating LLMs in Multi-Session Coding
  Interactions
Authors: name: Nathanaël Carraz Rakotonirina; name: Mohammed Hamdy; name: Jon Ander Campos; name: Lucas Weber; name: Alberto Testoni; name: Marzieh Fadaee; name: Sandro Pezzelle; name: Marco Del Tredici
Published: 2025-02-19 14:58:04

Abstract: Large Language Models (LLMs) are increasingly used in working environments
for a wide range of tasks, excelling at solving individual problems in
isolation. However, are they also able to effectively collaborate over
long-term interactions? To investigate this, we introduce MemoryCode, a
synthetic multi-session dataset designed to test LLMs' ability to track and
execute simple coding instructions amid irrelevant information, simulating a
realistic setting. While all the models we tested handle isolated instructions
well, even the performance of state-of-the-art models like GPT-4o deteriorates
when instructions are spread across sessions. Our analysis suggests this is due
to their failure to retrieve and integrate information over long instruction
chains. Our results highlight a fundamental limitation of current LLMs,
restricting their ability to collaborate effectively in long interactions.