Only add/remove tags if tags array is not empty.