It is dependent on the privacy settings that you have selected. If the privacy setting of your story is changed from 'My friends' to 'Everyone,' then they will be able to view your story.