Remove some unnecessary casts
ArrayRef-ize some ctor initializer related APIs