Fix the prefix guard on directory removal

Reported by:	kib
Fixes:		#1095
As soon as we were not able to remove on directory stop trying to remove parents

Related to bug:	#1095