Obama said the visit to Robben Island made him realize that everyday worries in the U.S. are " fairly trivial " compared to the struggle Mandela and other inmates went through .