refactor loss-augmented inference again, remove hack. this should be the right interface hopefully