Do not build torch for all devices