SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The amalgamation of multiple models, often termed as "ensemble or reassembly," can significantly enhance model performance [35]. The efficacy of the attack is potentially contingent upon the performance of the classifiers. Notably, even when the stable diffusion model generated the desired subjects with precision, there remains a possibility that the classifier may erroneously classify it. To eliminate the impact of the performance of a single classifier, we design a voting mechanism by using an ensemble of multiple classifiers. To be specific, the attack is defined as a failure if at least one classifier in the ensemble still recognizes the original categories in the text prompt. When we took this mechanism to evaluate the generated image after our attack, the success rate of the attack on the long and short prompts remained the same,  $81.8\%$  and  $91.1\%$  respectively. This further proves the effectiveness of our algorithm. Additionally, we test the voting mechanism with images generated from clean prompts covering the 1000 classes in ImageNet. As shown in the Table 5, most of the classifiers have a relatively high Top1 accuracy. We conducted experiments with three distinct classifiers: CLIP+VIT-B/16, Swin-B, and the vanilla VIT-B/16. The classification accuracies of these models were evaluated using both long and short text prompts. For the CLIP+VIT-B/16 model, the classification accuracies with long and short text prompts were found to be  $78.80\%$  and  $82\%$ , respectively. The Swin-B model demonstrated classification accuracies of  $76.20\%$  with long text prompts and  $79.80\%$  with short ones. Lastly, the vanilla VIT-B/16 model achieved classification accuracies of  $78.90\%$  and  $82\%$  with long and short text prompts, respectively. The probability that at least one of them successfully recognizes the current generated image is  $90.40\%$  and  $92.30\%$  for long and short text prompts, respectively, which is much higher than using only one classifier. This proves that our mechanism is effective.