Add guards for numpy and scipy