Cache found files instead of re-searching them

This should reduce the number of API requests quite a bit.