If he only ever calls or messages you late at night, there's a chance that he only wants you around for a physical relationship.