Using Kcachegrind has been clarified that swap and compare are the two major points of actention.
After a change to swap the timing has improved quite a bit.