BUG: disallow mixed dtype operations

closes #5198