Only remove fences and signs from inventory if expected