# Quantitative Characterization of Complex Systems--An Information Theoretic Approach
## 3 Understanding Entropy--An Overview  
The concept of entropy is often thought of as abstract and, at the same time, difficult to present because of its different applications by various authors in many research fields. Throughout the literature, it is observed that many authors mostly seem to consider entropy to be either a state of disorder or a loss of information. To better understand the distinct sense of the term entropy, a principled polysemy approach [10] adapted from the field of linguistics will help the readers to understand the basic co-existence of many meanings of the word entropy. Five distinct meanings of the word entropy were identified by [PERSON] et al. [10], viz., thermodynamic sense, statistical sense, disorder sense, information sense, and homogeneity sense. In the thermodynamic sense, entropy can be interpreted as a state of a system that tends to increase towards a maximum value. The statistical sense of entropy presumes a probabilistic approach of identifying, counting, and monitoring the transitions between the microstates of the system. In contrast to the statistical sense, the  
Figure 1: Agnostic two-stage complex systems examination structure. disorder sense does not use a probabilistic approach but rather considers the snapshot of a given situation by representing spatial configurations and messiness. Entropy in terms of information theory can be seen as the average rate of information added by the next element, calculated by considering the complete set of symbols and their probabilities. The information sense of entropy, according to [PERSON] et al., relates to the information needed to produce or interpret a message using its elements, such as digits, symbols, letters, and words. This entropy model shares the relationship between a message and its constituent elements. To better understand the information sense of entropy, refer to the following text extracted from [PERSON]'s work on mathematical theory of communication [10]:  
_\"If a source can produce only one particular message its entropy is zero, and no channel is required. For example, a computing machine set up to calculate the successive digits of \(\pi\) produces a definite sequence with no chance element\"_[11; 12]. Contrary to the statistical sense, where characteristics are shared by microstates, entropy in the information sense can be used to predict the next or an upcoming message or a symbol in a message based on conditional probabilities, thereby stressing the probabilistic prediction of the next symbol of a message. The main difference emphasized here is the characteristic of entropy according to the statistical sense based upon a system's description through its constituent elements' relationships. The information sense and the disorder sense share this property, whereas the thermodynamic sense does not. Entropy in thermodynamics represents the inherent disorder in a system over a period of time as the system heads towards thermodynamic equilibrium. In information theory, entropy helps to quantify the information [11]. Quantifying information implies analyzing the information present and measuring its associated uncertainty. Higher values of entropy signify lesser order in a system and lower values of entropy signify a more ordered system. Shannon entropy \(H\) is given as:  
\[H=-K\sum P_{i}\;\log_{2}\;P_{i} \tag{1}\]  
where \(K=1\), \(P_{i}\) is the probability of a symbol appearing in a given stream of symbols, and the use of the logarithm base two corresponds to expressing information entropy in terms of bits. This paper takes its foundation for entropy from information theory, as given by [PERSON]. Information theory plays an important role in understanding the meaning of entropy. This is because it is preferable to consider information that can be defined both quantitatively and subjectively. Moreover, information can be used as an anchor to understand what is changing in a spontaneous process.