FIX: core: unchecked realloc

Also changed types to Py_ssize_t, and check for overflow when multiplying.