On the Mistaken Assumption of Interchangeable Deep Reinforcement Learning Implementations
Creators: name: Singh Hundal, Rajdeep
name: Xiao, Yan
name: Cao, Xiaochun
name: Dong, Jin Song
name: Rigger, Manuel
Publisher: name: Zenodo
Year: 2025

description: This is the artifact (source code) for our paper titled "On the Mistaken Assumption of Interchangeable Deep Reinforcement Learning Implementations", where through a differential testing lens, we tested different Deep Reinforcement Learning (DRL) implementations of the same algorithm. This artifact contains the necessary scripts to easily replicate the literature review and experiments from our paper, along with detailed instructions on how to do so.For context, firstly, we conducted a literature review where we automatically (1) downloaded accepted conference papers from Neural Information Processing Systems (NeurIPS) and the International Conference on Machine Learning (ICML), and (2) filtered through them by using keywords to determine if there were studies that made the assumption of interchangeable algorithm implementations. Secondly, we conducted a pilot study to train and compare five Deep Q-Network (DQN) implementations across three environments. Thirdly, we conducted a study of larger scale to train and compare five Proximal Policy Optimization (PPO) implementations across 56 environments. Lastly, we replicated three experiments from a study, but with a different Deep Deterministic Policy Gradient (DDPG) implementation. This was done to determine if using a different implementation was significant enough to flip experiment outcomes.As our experiments involved training 1520 DRL agents (10K computational hours), we have also included all of our training outputs as well to facilitate directly using state-of-the-art DRL comparison techniques on our outputs to reproduce the figures and tables presented in the paper. This allows you to bypass training all 1520 DRL agents. We recommend this approach since training all agents can be computationally expensive. We have provided a docker image to automate the recommended approach, which includes performing the literature review as well. However, we have also provided the necessary scripts to train the agents from scratch. Using this artifact, you will be able to replicate (1) the literature review, (2) the pilot study where we tested DQN implementations, (3) RQ1 where we tested PPO implementations, (4) RQ2 where we debugged PPO implementations, and (5) RQ3 where we replicated a study with a different DDPG implementation.This artifact has two zip files; non-lightweight.zip and lightweight.zip. The non-lightweight.zip file consists of the full source code which includes our training outputs as well as two docker images; one to automate the recommended approach and another to demonstrate how an agent is trained on your machine (a GPU is not needed here). The lightweight.zip file excludes our training outputs and the docker images and should be downloaded if you intend to train the agents from scratch or inspect the source code.

Setup requirements for the recommended replication approach (with a training demo):



25 GB Storage

Docker installed and configured appropriately

Download non-lightweight.zip


Setup requirements for training all 1520 agents (~10K hours):



At least four mid-range NVIDIA GPUs

A CPU with at least 32 cores

30 GB Storage

Ubuntu OS

Bash, Tmux, Git, chmod, Anaconda, and CUDA installed and configured appropriately

Download either non-lightweight.zip or lightweight.zip