Unfortunately, yes. Even clean retainers can cause cavities. When you wear a properly fitted retainer, it can trap bacteria next to your teeth which can cause cavities (source).
Bacteria on your retainer can also contribute to tooth decay and cavities. That’s why it’s important to minimize that bacteria and eliminate some risk of cavities.