I can't handle it. It certainly is true that one doesn't have to have a traumatic experience to acquire a specific phobia or intense chronic anxiety. Yet, as already mentioned, basic learning principles could produce a phobia or serious anxiety without a painful trauma being involved. Avoiding approaching interesting people for years can make it too scary to do. No, it's probably the other way around.
Most people simply have a good idea of how well they can handle a situation.
Science has not yet explained why and how exaggerated threats are learned and combined with fluctuating estimates of our self-efficacy in one specific phobic situation. It truly seems that stress for most people comes primarily from the person's own personality or general nature, i.e. they are the type, often with low self-esteem, that respond more intensely to environmental stresses that are common in everyone's life. Psychology is not doing a good job in this area, as yet. Thus, the level of insecurity differs from person to person. Some of us deny problems, while others exaggerate problems.