Unless that particular port 22 is exposed to the Internet, directly or indirectly, you have nothing to worry about. If it is, set up public key authentication, disable password authentication, and don't worry about it.