You should make every possible effort to recollect and report this information. Its absence could seriously slow the process of your appointment. The FBI recommends that I pick references that live in the same area of the country to speed things up. In such a case, the FBI will probably want to interview references in every location where you've lived. You can speed up the full-field investigation by providing as many names as possible for each area of former residence.
Presidential appointees serve at the pleasure of the president.
They may be removed by the President, but not by the Congress. Congress may, however, impeach and seek to convict any officer of the government. A successful impeachment of a presidential appointee in the executive branch has never occurred.