Yes, the SSH server is the only one that can start any network connection. Additionally, it keeps an eye on new relationships and reacts to them. The servers do that by using TCP port 22 on the host machine.
Every SSH connection requires the server to use public key cryptography to identify itself to the client at the outset. Client authentication is required so the SSH client may confirm that it is a trusted server, not an intruder.