Multilingual-cased on Portuguese prompts?
Did you test the multilingual-cased model on Portuguese prompts? The external test set is deepset/prompt-injections, which is almost all English and German. I would like to know whether the out-of-distribution drop changes when the injection is written in pt-BR.
Thanks for the question! I haven’t looked at Portuguese yet, so I can’t say whether the drop changes in pt-BR. It’s a great direction, though, and one I’d like to explore: a small hand-written pt-BR evaluation set is something I want to build, and I’ll share the results here when I do. If you know of a public pt-BR prompt-injection dataset, I’d be glad to take a look.
Thanks for the answer. A small hand-written pt-BR evaluation set sounds like the right way to test it, and I will be glad to read the results when you share them.