As @astrofrog suggested please definitely break up this test into, I'm thinking three?  One for each integer size case?

You could write most of this test just once and then parametrize it.  See for example https://github.com/astropy/astropy/blob/master/astropy/io/fits/tests/test_image.py#L496

(I realize there are some special cases like in the 64-bit case but that can just be handled in an if-statement.)