This will ultimately depend on your needs. Generally speaking, the size of a unit should not affect its performance so it will be up to you to decide if the size does not matter or if you want the smallest air conditioner in the market.
But if you have a small space, you should consider that some larger units may be difficult to store away or even move around. And if you have a large space, you may worry that portable air conditioners on the smaller side may not be able to cool your large space as fast as a bigger one might be able to.