Her mother was consigned to a remote apartment in Buckingham Palace, and Victoria often refused to see her