Don't make a redundant copy when getting double from buffer, that may cause alignment issue on ARM platform