manifest: Drop libinput from rebuild

I can't figure out how to specify it properly