Relying heavily on a draft model for filtering out prompt candidates may introduce certain limitations or drawbacks. One potential issue is that the draft model may not capture all the nuances and complexities present in the target model, leading to inaccurate filtering decisions. This could result in discarding potentially valuable prompt candidates that might have been effective with the target model. Additionally, over-reliance on the draft model may limit the exploration of diverse prompts and hinder creativity in generating adversarial examples.