BUG: Bug in downcasting inference with empty arrays (GH6733)

closes #6733