[WIP] Optimize the solver and remove unnecessary code

After the recent changes in #2733 we can optimize the solver a bit