Refactor and cleanup inference, leaving only geo for now