## Appendix A Full Questionnaire (Updated Version--60 Questions)
### Test Debt--Questions 48 to 54  
Question 48: Have the hyperparameters been properly tuned and validated to ensure optimal performance within the game environment? Stakeholder: Participants  
Score: 5  
Justification: Tuning hyperparameters directly affects the performance of the RL agent in the game environment. It's crucial for optimizing its behavior. Optimal hyperparameters are vital for the RL agent to effectively learn and adapt to the game dynamics. Correct hyperparameter tuning is essential in RL game applications to optimize the model's performance and enhance the gaming experience, ensuring efficient learning and effective decision-making in-game scenarios. Example: Tuning the learning rate can significantly affect the convergence speed and stability of the training process. Question 49: Has reproducibility of agent training and environment dynamics been tested to ensure consistency? Score: 4 Justification: Reproducibility ensures consistency and reliability in training the agent, which is crucial for fair gameplay. Example: The use of fixed random seeds to ensure that results can be replicated across different runs. Question 50: Is there a fully automated test regularly running to validate the entire pipeline, ensuring data and code move through each stage successfully and resulting in a well-performing model? Stakeholder: Organizers Score: 5 Justification: Having a fully automated test ensures the integrity of the entire pipeline, which is essential for maintaining the quality of the game. Example: Automated regression tests can catch issues early before they affect the model's performance. Question 51: Do the data invariants hold for the inputs in the game environment? Stakeholder: Organizers and Participants Score: 3 Justification: Data invariants are important for maintaining the integrity of the game environment and ensuring fair gameplay. Example: Checking that the positions of players in a sports game do not suddenly jump to unrealistic values. Question 52: Are there mechanisms in place to ensure that training and serving are not skewed in the game? Stakeholder: Organizers and Participants Score: 4 Justification: Skewed training and serving can lead to unfair advantages or disadvantages for players. Example: Using the same feature engineering pipeline for both training and serving can help avoid discrepancies. Question 53: Are the models numerically stable for effective gameplay? Stakeholder: Organizers and Participants Score: 5 Justification: Numerically stable models ensure reliable and consistent behavior during gameplay. Example: The use of gradient clipping in training to prevent exploding gradients. Question 54: Has the prediction quality of the game not regressed over time? Stakeholder: Organizers and Participants Score: 5 Justification: Prediction quality directly impacts the agent's decisions and ultimately the gameplay experience. Therefore, maintaining its quality is crucial. Example: Tracking the accuracy of the model's predictions against a validation set over multiple seasons of a game. _Appendix A.18_. : _Versioning Debt--Questions 55 to 57_
* Question 55: Have you installed a proper version control system for model, training and test data? Stakeholder: Organizers Score: 5 Justification: Version control is crucial in any software development project, including RL game applications, to keep track of changes, revert to previous states if necessary, and collaborate effectively. Without version control, it can be challenging to manage changes, leading to potential errors and difficulties in reproducing results. Example: If a new model version causes a regression in performance, a version control system would allow developers to quickly revert to a previous, stable version. Question 56: Have you used the appropriate policy for marking the versions of your software components? Stakeholder: Organizers  
Score: 3  
Justification: Proper versioning allows for clear communication about changes and updates to software components.