You're safe, as it's not possible. They won't know you're preventing them from seeing your Stories unless they ask a common friend to check or use another account to infiltrate your friends list.