I can surely understand how you feel.  And although security should have double checked with you and made sure he had legal access, the probably relied on the past pattern.  What I would do is to speak with the head of security and advise them that he is no longer in the picture and that no one is to have access to the apartment with out your written consent. Make sure that the head explains it to every shift. And if you think that he might have a key change the locks.  Good luck.