Remove libraries that were not found on sync

It might be a good idea to check out what happens if a library is added
back.