Don't prevent destroying fragments.

It might lead to multiple fragments being incorrectly attached; and we
prevent unneeded destroyings via getItemPosition() already.