Remove unnecessary unsafe.Pointer casts