Use non-destructive API for applying input adapters

The API seems that it doesn't change destructively but it changes
destructively for now. See added comment about TODO.