Statistics: Outlier Data

I am glad we share a love of statistics! I am looking forward greatly to this debate. For this debate, significance will be defined as the amount of focus and effort given in consideration to normal data/outcomes or outlier data/outcomes. Here is an example (using micro data. Therefore this example does not fit the resolution): John sleeps 99% of all nights. The outcome of normal data is that he needs a place to sleep. In designing his sleep system, he considers this outcome first and foremost, and implements a bed into his system, in order to garner an outcome that yields the highest net benefits (comfortable, resting sleep). We can see, that in this case, we can see that in giving normal data high significance when designing this system, he has created an effective system (albeit simple). Pros CaseThe inspiration for this debate came to me after reading excerpts from The Black Swan by Nicholas Teleb. The introduction of the book tells the story of the discovery of black swans. For a very long time, scientists all assumed that all swans were white. It wasn't until a statistical outlier was discovered that this assumption was rocked and re-made. Obviously, this is not a case of statistics where a traditional system is artificially designed, but the point stands. In this debate, I will typically refer to events/data points that are statistical outliers as "Black Swan Events" as an homage to the book. I. Big Data, and Outliers(A) Big Data means countless "trials. "There are many systems in the modern world that go through countless numbers of "Trials" daily. Any action that the majority of people do daily, or at least often, will go through countless numbers of trials. For example: Driving. Millions of driving trips are taken every day. Each one of these trips can be considered a trial. The outcomes are categorical, and numerous, but most can be lumped into two outcomes "Safe Trip" and "Unsafe Trip. " I'll talk more about this example in part II. (B) The Unlikely is Bound to Happen. When you have countless numbers of "trials" occuring on any kind of action, no matter how unlikely a certain outcome may be, it is going to happen. For example, Trial A has a 99.999999% chance of resulting in outcome TRUE and a 0.000001% of resulting in outcome FALSE. In terms of micro-data, if you ran 100 trials, you'd probably only get TRUE outcomes. Therefore, you should probably design your system with only the implications of TRUE in mind. However, if 300,000,000 people perform the relevant action to Trial A twice a day, suddenly you have 4,200,000,000 trials. This means that you will likely get 42 FALSE outcomes. I am arguing that, in this case, systems designed around Trial A should be designed under the assumption that outcome FALSE will eventually occur. These are "Black Swan" events. II. Examples of "Black Swan" events(A) 9-11 Terrorist AttackThe odds of being in a plane crash are 1 in 11,000,000 (1). An infintesimly small percantage of flights end in crash due to terrorists. Considering, though, that there are around 10 million flights every year (2), it's not too surprising that the unlikely did occur. As I discussed in Point I, Subpoint B, the unlikely eventually happens. Obviously, 9-11 was a statistical outlier. If flight systems had been designed with this outlier in mind, the attacks could have possibly been prevented. Preventative systems on planes would have been a small price to pay to prevent the September 11th attacks. (B) Car SafetyAnother excellent example of outlier data significance is car safety. Daily, there are about 200 million people who drive daily (3). As most people take at least 2 driving trips per day, we can state that there is a minimum of about 400 million driving trips taken daily. There are 16,900 car wrecks daily (4). This means that about 0.004225% of driving trips end with the outcome I am labeling WRECK. Although our regular data results in outcome SAFE TRIP, we design our car system with a very large significance attributed to the WRECK outcome. We add seatbelts, airbags, and other safety systems, and we drive more carefully. We do this in order to both reduce the probability of a wreck, and to prepare ourselves for the unlikely. (C) Investing ModelsIn investing, the traditional investing strategy is to build you portfolio on what is most likely. In Teleb's book Anti-Fragile Teleb outlines his investing strategy based on what is unlikely. He describes in his book his three kinds of investing systems. These systems are either fragile, robust, or anti-fragile. A fragile system, when put under stress, will fail. A robust system when put under stress, will remain strong, or falter slightly. An anti-fragile system, when put under stress, will thrive or improve. Teleb encourages creating a portfolio comprised only of robust and anti-fragile systems. This is because although extreme stressors (such as recessions) are incredibly unlikely, because of the volume of trials, these stressors are bound to happen. If you account for these stressors when designing your investing system, when they do eventually pop-up, you will make much more money than someone who invests only in fragile systems. (D) Medicine, and False PositivesIn medicine, false positives are a very common problem in testing for illnesses. Suppose that an illness occurs in 1 in 1000 people (0.1%). There is a test for this illness that gives a false positive only 1% of the time. Having a 1 in 1000 illness is a statistical outlier, but the system is not designed to account for this. Because 1 in 100 people are given false positives, and only 1 in 1000 have the illness, that means that if you tested 1000 people for this illness, there would be 9 false positives and 1 legitimate positive. If the system was refined for this statistical outlier, you would have a much lower rate of false positives (in this case, 90% of all positives being false). Sources1. . http://www.artofmanliness.com...2. . http://www.cnn.com...3. . http://www.ask.com...4. . http://www.ask.com...;