A localization framework that works end-to-end