During the second half of the 19th century and the first half of the 20th century , it gradually added neighboring islands and territories , most with Greek-speaking populations .