build: clean: Fix empty module infinite loop issue

Signed-off-by: Bart Kuivenhoven <bemkuivenhoven@gmail.com>