Yes, a statement made to only a single other person can be defamation, at least in the US (you don't mention the jurisdiction that you or the accused person are in, and it may matter). Only the accused person can normally sue, and that person would need to establish that the statement was made, and that it was false.