You can set up a whitelist that includes only your friends. Alternatively, disable port forwarding and keep the server’s IP address private among your friends.