Only do *&ex -> ex optimisation if there is no type punning involved