Only destroy sortables if they have existed

Avoids an error being thrown while setting up list items