Although the answer to this is highly subjective, most men only allow you to touch their hair or beard if you're really close or they like you a lot. So if he lets you touch your beard and play with it, chances are you mean a lot to him.