Q:

Why did I lose 20 rep? 

I just "lost" 20 rep, at my profile summary it says "User was removed", what is that? 
a screen capture:

A:

These votes correspond to upvotes from a user who now deleted their account.  Account deletion undoes all actions of that user.  Consequently all they upvotes are removed.