serial: remove dpll cacading

Try to fix sign lost when suspend

Change-Id: I7fcdf93de9e09f333704f856f78e601412c61728