# AlgoAce: Retrieval-Augmented Generation for Assistance in Competitive Programming

Anav Agrawal, Jill-Jenn Vie

► To cite this version:

Anav Agrawal, Jill-Jenn Vie. AlgoAce: Retrieval-Augmented Generation for Assistance in Competitive Programming. CSEDM 2025 - 9th Educational Data Mining in Computer Science Education Workshop, Jul 2025, Palermo, Italy. 10.1145/nnnnnn.nnnnnnn. hal-05089333v2

HAL Id: hal-05089333

https://hal.science/hal-05089333v2

Submitted on 27 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# AlgoAce: Retrieval-Augmented Generation for Assistance in Competitive Programming

Anav Agrawal

Jill-Jênn Vie

anavagrawal.iitd@gmail.com

jill-jenn.vie@inria.fr

Soda Team, Inria Saclay

Palaiseau, France

## Abstract

Competitive programming platforms like Codeforces are widely used for self-directed learning and assessment in computer science education. However, students often struggle to identify relevant practice problems or receive targeted advice without personalized guidance. In this paper we present AlgoAce, an AI-powered assistant that integrates with Codeforces and uses retrieval-augmented generation (RAG) to provide support while attending at two key sources of information: user submission history, and problem metadata. The system leverages large language models to fetch relevant problems and past user submissions, grounding user queries in retrieved content before generating responses. AlgoAce adapts to user preferences—such as programming language or problem tags—and supports reflective learning through natural language interactions. We describe the tool’s architecture and discuss its potential for integration into the competitive programming learning ecosystem.

## CCS Concepts

* Applied computing → Computer-assisted instruction; Interactive learning environments; Computing methodologies → Natural language generation; Information systems → Recommender systems; Question answering.

## Keywords

Retrieval-augmented generation, Competitive programming, Intelligent tutoring systems, Codeforces

## ACM Reference Format:

Anav Agrawal and Jill-Jênn Vie. 2025. AlgoAce: Retrieval-Augmented Generation for Assistance in Competitive Programming. In *Proceedings of AIED GenAI workshop (AIED GenAI 'XX)*. ACM, New York, NY, USA, 6 pages. https://doi.org/10.1145/nnnnnnn.nnnnnnn

## 1 Introduction

Competitive programming platforms, such as Codeforces<fnref n="1" />, HackerRank, or Kattis, provide thousands of problem-solving challenges.

They primarily focus on ranking programmers by organizing programming contests, but they are also used by participants for practice and self-directed learning. However, it is always easier to rank students than to upskill them: surprisingly, little effort has been directed towards building tools to improve the skills of competitive programming contestants. Students frequently struggle to identify exercises that align with their current skill levels. There remains a gap in providing personalized, actionable feedback on what students should learn or practice next.

Recently, Large Language Models (LLMs) have emerged as powerful tools in educational applications, but they often face challenges related to hallucinations—generating incorrect or misleading information. To mitigate this issue, Retrieval-Augmented Generation (RAG) was introduced [17]. RAG enhances LLM outputs by incorporating relevant context retrieved through nearest neighbor search in a vector database, thereby improving the accuracy and relevance of the generated responses.

In this paper, we introduce AlgoAce, an LLM-powered assistant designed to recommend relevant Codeforces problems to users based on their submission history, using a RAG architecture. A demo video is available<fnref n="2" />, as well as the code<fnref n="3" />.

## 2 Related Work

### 2.1 Large Language Models in Programming Education

The integration of LLMs into programming education has demonstrated remarkable potential across various applications [15, 26]. Previous work has explored LLMs for automated code generation at competitive levels, with systems like AlphaCode achieving performance within the top 54% of programming competition participants [12, 18], or even achieving gold medal level at the prestigious International Olympiad in Informatics (IOI 2025) live [8]. Beyond code generation, LLMs have been applied to knowledge tracing and learner performance modeling, showing capability in predicting student behavior and learning trajectories [24].

In programming practice environments, LLMs have been used to provide personalized guidance by analyzing learning styles and adapting to individual programming patterns [14, 20]. The task of knowledge tracing is to predict student performance based on history of attempts [7]. While knowledge tracing data was initially merely composed of a sequence of binary data of successful and unsuccessful attempts of exercises, several deep learning models

<figure><img src="image_2.png" /></figure>

This work is licensed under a Creative Commons Attribution 4.0 International License. AIED GenAI 'XX, Palermo, Sicily, Italy

© 2025 Copyright held by the owner/author(s).

ACM ISBN 978-1-4503-XXXX-X/2018/06

https://doi.org/10.1145/nnnnnnn.nnnnnnn

<footnote n="1">¹https://codeforces.com/</footnote>

<footnote n="2">²Demo video: Google Drive link</footnote>

<footnote n="3">³Source code: https://anonymous.4open.science/r/AlgoAce-D138/README.md</footnote>
<figure><img src="image_3.png" /><figcaption>Figure 1: RAG pipeline used by AlgoAce</figcaption></figure>

have been developed that have access to the submission code of participants [19, 28]. Having access to this data can also help detecting bugs in student code, using approaches like code2vec or ASTNN [29], or having two LLMs interacting with each other to suggest changes and implement them [25].

## 2.2 Retrieval-Augmented Generation for Educational Applications

Retrieval-augmented generation (RAG) has shown promise in improving the quality and relevance of educational outputs by grounding LLM responses in external knowledge [10, 11, 22]. In programming education, RAG has been successfully applied to improve feedback generation by incorporating lecture transcripts in order to reduce hallucinations [13]. Mathematical education has also benefited from RAG approaches, where textbook content is retrieved to enhance question-answering systems for middle-school students [16]. Recent studies have explored RAG-based course recommendation systems, such as RAMO [27], which leverages course metadata and descriptions to provide personalized MOOC recommendations. However, RAMO does not personalize course description from other data sources beyond the user query.

## 2.3 Personalized Exercise and Problem Recommendation

Personalized exercise recommendation in programming contexts has seen limited exploration compared to broader course recommendation systems. Some works have applied reinforcement learning techniques to exercise recommendation, using knowledge tracing models to predict student performance and suggest appropriate practice problems [2, 3, 5, 31]. Explainable recommendation systems have been developed for programming practice environments, providing justifications for recommended activities based on student knowledge states [4, 9]. However, these systems typically operate on coarse-grained course-level data rather than the fine-grained

problem-level personalization required for competitive programming environments.

AlgoAce addresses this gap by incorporating users' historical problem attempts and performance data into its recommendation pipeline. This enables the system to dynamically adapt to a learner's evolving skill set, offering targeted practice that goes beyond traditional course-level recommendations to provide problem-specific guidance grounded in both semantic similarity and individual learning patterns.

# 3 Our RAG solution AlgoAce

## 3.1 Data Sources

AlgoAce operates on two primary data sources to provide exercise recommendations.

*Codeforces Problemset.* The first data source comprises the currently available problems on Codeforces, fetched via the official Codeforces API. Each Codeforces problem is represented by the following metadata: its title, associated tags (e.g., "shortest paths"), Elo rating, and URL, see an example in Table 1. The difference in Elo rating between a user and a problem is proportional to the logit of the probability that the user can solve the problem [1]. Since the Codeforces API does not provide full problem statements, AlgoAce embeds this available metadata into a dense vector space to enable semantic retrieval.

*User Submission History.* The second data source comprises past user submissions obtained when users input their Codeforces handle. The tool fetches the complete submission history of the user through the Codeforces API. The user submission history contains metadata of the attempted problem and the result of the submission (OK, Wrong Answer, or Time Limit Exceeded), see an example in Table 2. It is similar to the data encountered in knowledge tracing setting. Using this, AlgoAce recommends exercises of appropriate difficulty while targeting shortcomings.
<table><caption>Table 1: Example of Codeforces problems</caption><thead><tr><td>Rating</td><td>Name (URL)</td><td>Tags</td></tr></thead><tbody><tr><td>1700</td><td>Mathematical Problem</td><td>brute force, constructive algorithms, geometry, math</td></tr><tr><td>1200</td><td>Training Before the Olympiad</td><td>constructive algorithms, games, greedy, implementation, math</td></tr><tr><td>1000</td><td>Two Divisors</td><td>constructive algorithms, math, number theory</td></tr><tr><td>800</td><td>2023</td><td>constructive algorithms, implementation, math, number theory</td></tr><tr><td>1800</td><td>Bicycles</td><td>graphs, greedy, implementation, shortest paths, sortings</td></tr></tbody></table>

<table><caption>Table 2: Example of user submission history</caption><thead><tr><td>Rating</td><td>Verdict</td><td>Name (URL)</td><td>Tags</td></tr></thead><tbody><tr><td>2300</td><td>OK</td><td>Farm Game</td><td>combinatorics, games</td></tr><tr><td>1900</td><td>Timeout</td><td>Yet Another Permutation Constructive</td><td>*special, constructive algorithms</td></tr><tr><td>1900</td><td>OK</td><td>Yet Another Permutation Constructive</td><td>*special, constructive algorithms</td></tr><tr><td>2000</td><td>Wrong</td><td>Narrow Paths</td><td>*special, combinatorics</td></tr><tr><td>2000</td><td>OK</td><td>Narrow Paths</td><td>*special, combinatorics</td></tr></tbody></table>

## 3.2 Prompt Engineering and Structure

To enhance the effectiveness of retrieval and the quality of generated responses, AlgoAce employs a two-stage prompt engineering pipeline consisting of (1) an LLM-driven query rewriting stage and (2) a response generation stage.

### 3.2.1 Query rewriting for retrieval.
In this stage, the user's original query is first passed through a dedicated prompt that instructs the language model to reformulate the query using concrete terms aligned with problem metadata—such as problem tags (e.g., greedy, dp, graphs), verdicts (e.g., WRONGAnswer, OK), difficulty ratings (e.g., 800, 1000, 1700), or specific problem names. The prompt template for the query rewriting stage is provided in Appendix A.1.

This approach addresses a well-known challenge in retrieval-augmented systems: the semantic gap between how users naturally phrase queries and how information is indexed or tagged in the database. By explicitly rewriting queries to include relevant keywords and metadata terms, we greatly improve the likelihood that the retriever will surface the most relevant documents or problems [21].

### 3.2.2 Response generation.
After retrieval, the set of candidate problems is provided as context to the language model, along with the user's original query. AlgoAce employs a prompt template that leverages few-shot prompting techniques to guide the language model's responses. Few-shot prompting, where the model is provided with a small number of examples to demonstrate the desired behavior, has been shown to significantly improve performance on various tasks without requiring parameter updates [6, 23].

The prompt template for response generation is provided in Appendix A.2. The prompt incorporates two key techniques: few-shot prompting through examples that demonstrate the expected response format, and chain-of-thought reasoning triggered by the /think directive at the end. This /think token encourages the model to engage in explicit reasoning about the user's submission patterns before generating recommendations, leading to more thoughtful and well-reasoned responses [32].

## 3.3 Retrieval-Augmented Generation Architecture

The Retrieval-Augmented Generation (RAG) architecture consists of two main components: a retriever that identifies relevant information from the knowledge base, and a generator that produces responses based on both the retrieved context and the input query.

AlgoAce employs two separate retrievers operating in parallel. The first retriever focuses on fetching relevant user submissions from the submission history, identifying patterns in past performance and areas of difficulty. The second retriever searches through the available Codeforces problems to find exercises that match the user's skill level and learning objectives. Both retrievers utilize semantic embeddings to perform similarity search, converting the textual metadata into multi-dimensional vectors using the embedding model.

The results from both retrievers are concatenated and fed into a structured prompt to guide the LLM's generation. Additionally, users can specify their preferred programming language, which is incorporated into the query to ensure language-specific recommendations and code examples. The system is designed with modularity in mind, currently supporting both API-based large language models such as OpenAI's GPT models and locally hosted open-source alternatives, so that the data never leaves the device. A diagram of the whole pipeline is also provided in Figure 1.

*Implementation Details.* AlgoAce is implemented using the Pathway framework, which provides efficient Retrieval-Augmented Generation (RAG) pipelines. For approximate nearest neighbor search, we use Pathway's KNNIndex class, which applies locality-sensitive hashing to perform fast similarity search across embedded problems. The user interface is built using Streamlit, with a Pathway RESTful API connecting the front-end to the backend services. All code from the platform is released on GitHub.
<figure><img src="image_4.png" /><figcaption>(a) Problem recommendations</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>(b) Suggested strategy</figcaption></figure>

<caption>Figure 2: Examples of AlgoAce's output for problem recommendations and strategies.</caption>

## 4 Evaluation

While recommender system evaluation is usually enabled by cross-validation on a dataset, using regression metrics such as mean squared error or ranking metrics such as recall, precision or normalized discounted cumulative gain, the recommendation of learning resources is more challenging. People learn over time, make multiple attempts, either successful or unsuccessful, and also the recommendations may be liked but not useful at all for improving learning gains: there is a disparity between user preference and actual skill improvement [30]. As for now, we conducted simple offline evaluation.

Figure 2 presents examples of outputs generated by AlgoAce. In the first example, the user asks, “What topics am I weak at?” and receives an analysis of their submission history, highlighting difficulties with topics such as binary search and dynamic programming, along with targeted practice recommendations. In the second example, the user asks, “What should be my learning path going forward?” and AlgoAce suggests a personalized progression, recommending specific problems in binary search, graphs, and dynamic programming based on the user’s recent challenges.

To evaluate the retrieval of submission history, we prioritized ensuring the relevance of retrieved documents. Our initial observations revealed a critical need for query reformulation: queries that didn't explicitly define terms like "OK" or "time limit exceeded" consistently yielded irrelevant submissions. This led us to implement a two-way prompting strategy, where the first prompt is specifically designed to reformulate the user's query, thereby improving the relevance of the retrieved submission history.

While we considered using a filter that relies on traditional database queries, the fact of relying on LLM token embeddings allows us to share information across tags.

## 5 Conclusion

In this paper, we presented an AI companion that relies on RAG for recommending programming exercises, see Figure 3 for the user interface. Compared to traditional LLMs, the particularity of our approach is to ground LLM generation on actual evidence of submission history and a database of algorithmic problems from Codeforces. We believe this is a next step towards interacting with a database in natural language, for personalized training.

Beyond what has been shown, our tool AlgoAce could be used as an authoring tool for a teacher or learner, either to assemble
<figure><img src="image_6.png" /><figcaption>Figure 3: The user interface of AlgoAce</figcaption></figure>

a diverse set of exercises for a practice contest, or to design new problems. Having access to the Codeforces database for generation allows to have accurate measurements of the difficulty of exercises, which can be used for recommending items within the zone of proximal development of students. AlgoAce could also be used to automatically generate plots and learning analytics related to student performance. We plan to explore those directions, which come with their own sets of challenges for evaluating our tool.

The fact that the backbone is a LLM allows natural explanations for the generated recommendations. The validity of those explanations should still be evaluated by students with various degrees of experience, and teachers. While at the time of submission, we could not conduct a user study, we plan to experiment AlgoAce on students for the preparation for programming contests at our university.

As future work, we plan to incorporate problem solutions into the retrieval and recommendation process. By analyzing official or community-provided solutions, we can better estimate the prerequisite skills required for each exercise, leading to more accurate and pedagogically meaningful recommendations. Additionally, we aim to align the objective of the LLM more closely with learning outcomes, ensuring that the system prioritizes educational progress rather than merely solving queries. This could involve fine-tuning prompts to emphasize skill development and problem-solving strategies.

## References

[1] Solmaz Abdi, Hassan Khosravi, Shazia Sadiq, and Dragan Gasevic. 2019. A multivariate ELO-based learner model for adaptive educational systems. In *EDM 2019-Proceedings of the 12th International Conference on Educational Data Mining*. International Educational Data Mining Society, 228–233.

[2] Prerna Agarwal, Avani Gupta, Renuka Sindhgatta, and Sampath Dechu. 2022. Goal-Oriented Next Best Activity Recommendation using Reinforcement Learning. arXiv:2205.03219 [cs.AI] https://arxiv.org/abs/2205.03219

[3] Fangzhe Ai, Yishuai Chen, Yuchun Guo, Yongxiang Zhao, Zhenzhu Wang, Guowei Fu, and Guangyan Wang. 2019. Concept-Aware Deep Knowledge Tracing and Exercise Recommendation in an Online Learning System. In *EDM*. International Educational Data Mining Society (IEDMS).

[4] Jordan Barria-Pineda, Kamil Akhuseyinoglu, Stefan Zelem-Celap, Peter Brusilovsky, Aleksandra Milicevic, and Mirjana Ivanovic. 2021. *Explainable Recommendations in a Personalized Programming Practice System*. 64–76. doi:10.1007/978-3-030-78292-4_6

[5] Stefano Branchi, Chiara Di Francescomarino, Chiara Ghidini, David Masimo, Francesco Ricci, and Massimiliano Ronzani. 2022. Learning to act: a Reinforcement Learning approach to recommend the best next activities. arXiv:2203.15398 [cs.AI] https://arxiv.org/abs/2203.15398

[6] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. arXiv:2005.14165 [cs.CL] https://arxiv.org/abs/2005.14165

[7] Albert T Corbett and John R Anderson. 1994. Knowledge tracing: Modeling the acquisition of procedural knowledge. *User modeling and user-adapted interaction 4*, 4 (1994), 253–278.

[8] Ahmed El-Kishky, Alexander Wei, Andre Saraiva, Borys Minaev, Daniel Selsam, David Dohan, Francis Song, Hunter Lightman, Ignasi Clavera, Jakub Pachocki, et al. 2025. Competitive Programming with Large Reasoning Models. (2025). https://arxiv.org/pdf/2502.06807

[9] Jibril Frej, Neel Shah, Marta Knezevic, Tanya Nazaretsky, and Tanja Käser. 2024. Finding Paths for Explainable MOOC Recommendation: A Learner Perspective. In *Proceedings of the 14th Learning Analytics and Knowledge Conference (LAK '24)*. ACM, 426–437. doi:10.1145/3636555.3636898

[10] Zifei FeiFei Han, Jionghao Lin, Ashish Gurung, Danielle R. Thomas, Eason Chen, Conrad Borchers, Shivang Gupta, and Kenneth R. Koedinger. 2024. Improving Assessment of Tutoring Practices using Retrieval-Augmented Generation. arXiv:2402.14594 [cs.CY] https://arxiv.org/abs/2402.14594

[11] Owen Henkel, Zach Levonian, Chenglu Li, and Millie Postle. 2024. Retrieval-augmented generation to improve math question-answering: Trade-offs between groundedness and human preference. In *Proceedings of the 17th International Conference on Educational Data Mining*. 315–320.

[12] Nam Huynh and Beiyu Lin. 2025. Large Language Models for Code Generation: A Comprehensive Survey of Challenges, Techniques, Evaluation, and Applications. arXiv:2503.01245 [cs.SE] https://arxiv.org/abs/2503.01245

[13] Sven Jacobs and Steffen Jaschke. 2024. Leveraging Lecture Content for Improved Feedback: Explorations with GPT-4 and Retrieval Augmented Generation. In *2024*
# A Prompt Template

## A.1 Query Rewriting Prompt

The query rewriting prompt is used to reformulate user queries with explicit metadata terms, improving retrieval accuracy:

You are a query rewriting assistant for a competitive programming tool.

The goal is to rewrite user queries so that they include concrete terms that match problem metadata stored in a database.

The database retrieves problems based on word overlap with the query - so include exact terms like problem 'tags' (e.g. 'greedy', 'dp', 'graphs'),

verdicts (e.g. 'WRONG', 'OK', 'RUNTIME_ERROR'), difficulty 'ratings' (e.g. 800, 1000, 1700), or specific problem names if known.

You do NOT have access to the user's past submissions, but you must guess the user's intent and include as many relevant keywords as possible to help the system match the query to relevant problems.

Your output should be in the following format:

```
Retrieval terms: <insert keywords for retrieval>
```

```
User query: <natural language query reformulated for the LLM>
```

```
Make sure the retrieval terms are rich in specific tokens, and the user query is clear and helpful for the LLM.
```

## A.2 Response Generation Prompt

The response generation prompt guides the LLM in analyzing user submissions and generating tailored recommendations or explanations:

```
You are a competitive programming coach.
```

```
I will give you the user code submissions and a list of all available problems after "Data:".
```

```
The data lines with verdict are the user code submissions.
```

```
The other lines are the available problems on codeforces.
```

```
Answer the user query after looking at the user code submissions and the questions he got a wrong answer in.
```

```
The problems you analyze and suggest should only be from the given data lines without verdict and should be similar in rating to the highest rated problems that the user has attempted.
```

```
The answer should also explain how you arrived at the answer looking at the user code submissions if necessary. Keep your thinking short.
```

```
Example query: "What problems should I practice next?"
```

```
Ideal Answer: "Looking at your submissions, you solve questions rated around 1500 and you are facing difficulty in implementing binary search.
```

```
So here are some binary search problems you can practice: [Links to Binary search problems]."
```

```
Example query: "How to implement dynamic programming?"
```

```
Ideal Answer: "Here's an implementation of dynamic programming. [Dynamic Programming Code]."
```

```
Data: \n{context} \n{query} /think.
```