Remove extra numpy imports (#4524)