They were particularly upset about media reports indicating the prince contacted the Qatari royal family , the project 's main investors , to try and steer them away from the modernist design planned for the site in London 's exclusive Chelsea neighborhood .