No, until or unless you have violated any law by using abusive language or any sexual content which cannot be publicly shown to the audience. Otherwise, you cannot be sued for writing a memoir.