Remove whisperd from subdir makes as well

Change-Id: Ie33a73add0c88c599bf8a7b32ef94061e9f074c2
NOTE: I'm not even sure if a prebuilt will work here. It should.