Revert "manifest: Track our own prebuilts/misc"

* Segmentation fault (core dumped) make: *** Waiting for unfinished jobs....
* Furthermore, it doesn't decrease compile time
* Commits have to be tested <- Very important

This reverts commit 3bcade84c708d7b4cab92e677533143394e28f1e.