Disallow taking access from users higher then you

Fixes #1