Yes, you can do the visit and just stay on the island while others swim. I don’t understand why you would want to miss out on this, but it’s perfectly fine.