The Roman Catholic leader made his comments to pilgrims gathered outside his summer residence in Castelgandolfo near Rome to hear his weekly blessing .