remove pkgconfig if removing headers, i.e. if not having I_USE_DEVEL

Signed-off-by: Mattias Andrée <maandree@operamail.com>
do not try to remove non-existing directories

Signed-off-by: Mattias Andrée <maandree@operamail.com>