Ruby Teaming: Improving Quality Diversity Search with Memory for
  Automated Red Teaming
Authors: name: Vernon Toh Yan Han; name: Rishabh Bhardwaj; name: Soujanya Poria
Published: 2024-06-17 15:36:14

Abstract: We propose Ruby Teaming, a method that improves on Rainbow Teaming by
including a memory cache as its third dimension. The memory dimension provides
cues to the mutator to yield better-quality prompts, both in terms of attack
success rate (ASR) and quality diversity. The prompt archive generated by Ruby
Teaming has an ASR of 74%, which is 20% higher than the baseline. In terms of
quality diversity, Ruby Teaming outperforms Rainbow Teaming by 6% and 3% on
Shannon's Evenness Index (SEI) and Simpson's Diversity Index (SDI),
respectively.