Discord's current functionality does not allow for custom visibility settings on a user-by-user basis. When you set your status to invisible, it is applied universally across all servers and direct messages.