Algorithm to find a solution of system with linear and quadratic equations