Revert "serial_core: Fix pm imbalance on unbind"

This reverts commit b040cc8640c291370a25f11e9da5a8e1840ce4bd.