Fix error when assigning NaN to an integer column

Also covers any non-castable case by returning nil, which
is in-line with the intention of the former implementation,
but covers the odd cases which respond to to_i but raise
an error when it's called, such as NaN, Infinity and -Infinity.

Fixes #8757

Backport of #8781

Cherry Pick of c147dd7ddd206f6c2d80d8957ee9fa575b00b13c