Fix the prefix guard on directory removal

Reported by:	kib
Fixes:		#1095