Although you could go purchase certificates from a public CA like Verisign, for our purposes a local CA is sufficient. Next you need to create a key and certificate for the server. The first openssl command will prompt you for a bunch of information. Otherwise most MUAs will warn the user that the server hostname doesn't match the name in the cert. You should now be able to restart Sendmail.
Check syslog for any errors.
Now the useful step, allowing users to relay if they present a certificate that you trust. That is the DN for your CA certificate. In other words, all of the elements have changed from being seperated by a comma and a space to being seperated by a forward slash. I did the conversion by hand, but you could whip up a Perl script if you have a lot to convert. There is support for some amount of regular expressions in the DN, see Claus' page for info on that.