Yes, you can delete the OpenSSH server and its configuration using the following command in Ubuntu.
sudo apt-get purge openssh-client