Yes, someone may remotely access your Mac without your knowledge since Macs are vulnerable to hacking. If SSH is enabled, anyone with administrator credentials can potentially execute code that could result in a total takeover of your Mac.