Although some families decided to return to Pitcairn in 1858 and 1863, the island's population continued to grow