Remove some unnecessary and complicated logic

Fixes #712