Clean up inference algorithm a bit