Diversity in complex systems: measuring parts of the distribution to the whole

Despite the utility of theHill numbers,measuring diversity within complex systems and complex datasets, particularly regarding parts of a distribution to thewhole—that is, different portions of diversity types to the entire systemor dataset—is a challenging issue. In this paper, we attempt to relate the diversity of a part (or parts) of a distribution to the diversity of thewhole.We derive this relationship for theHill numbers D q , and use these results to further examine the effect of a freely varying type on the diversity of thewhole distribution. 1.Measuring diversity As identified by Sornette (Sornette 2009) and others (Newman 2005, Barabasi 2009), probability distributions are the ‘first quantitative characteristics’ of both complex systems and complex datasets (Sornette 2009, p. 2). In terms of diversity (bothwith regard to species richness and evenness), thismakes themhighly useful, as measurements on awide range of complex systems and complex datasets arewell approximated by their shape, particularly as the sample size and its diversity  ¥ n . For example, within the complexity sciences, the literature onmeasuring diversity is vast and includes a number of differentmathematical formulations—GiniSimpson index, Shannon entropy,Hill numbers— (Jost 2006, Leinster andCobbold 2012, Chao and Jost 2015, Hsieh et al 2016, Pavoine andMarcon 2016, Jost 2018). TheHill numbers andRényi entropies are also important measures used to derive quantumuncertainty relations in quantum theory and also themaximal entropy principle in statistical inference (see Jizba et al 2015, Jizba andKorbel 2018 and references therein). Across thesemeasures, theHill numbers D q provide a highly useful way tomeasure the diversity of a distribution that computes the number of equivalent equiprobable types for a uniformdistribution that maintains the same level of entropy (Rényi entropy to be precise) (macArthur 1965,Hill 1973, Peet 1974, Jost 2006, Gaggiotti et al 2018, Jost 2018). The parameter q favours the types with lower frequencies if 0<q<1 and the types with higher frequencies for q>1. For q=1, D 1 weights each type proportional to their relative frequency of occurrence, and boils down to ewhereH is the Shannon entropy of the distribution. Still, despite the utility of theHill numbers,measuring diversity within complex systems and complex datasets, particularly regarding parts of a distribution to thewhole—that is, different portions of diversity types to the entire systemor dataset—is a challenging issue. In this paper, we attempt to relate the diversity of a part (or parts) of a distribution to the diversity of thewhole.We derive this relationship for theHill numbers D q , and use these results to further examine the effect of a freely varying type on the diversity of thewhole distribution. Our interest in this issue is based on a series of papers we havewritten on diversity in complex systems and complex datasets, inwhichwe have introduced amodification of D 1 called case-based entropy Cc tomore effectivelymeasure the true diversity of some dataset of study, including the restriction of diversity within complex systems (Rajaram andCastellani 2012, 2014). As ameasure,Cc is based on amodification of the Shannon-Wiener entropyH; and is inspired by Jost’s paper onEntropy andDiversity, (Jost 2006), inwhich he states that: ‘In physics, economics, information theory, and other sciences, the distinction between the entropy OPEN ACCESS RECEIVED 9March 2020 REVISED 24March 2020 ACCEPTED FOR PUBLICATION 30March 2020 PUBLISHED 16April 2020 Original content from this workmay be used under the terms of the Creative CommonsAttribution 4.0 licence. Any further distribution of this workmustmaintain attribution to the author(s) and the title of thework, journal citation andDOI. © 2020TheAuthor(s). Published by IOPPublishing Ltd of a system and the effective number of elements of a system is fundamental. It is this latter number, not the entropy, that is at the core of the concept of diversity in biology’ (p. 363). To our knowledge such an approach, relative to the probability distributions of complex systems or complex datasets in general (particularly big data), has not been derived or explored before. As such, for us, Jost’s quote— and, in turn, our development of Shannon-Wiener entropyH intoCc—provides one potentially useful way for advancing the statisticalmeasurement of diversity within complex systems in general. In addition, we anticipate the use of our results in furthering our development of case-based entropy and deriving newproperties of the same. Remark 1.1. Wenote that the choice of q actually encodes the kind of correlation that exists between the parts of Pi of the distribution (Jizba andKorbel 2018). = q 1means that the parts are independent and the strong system independence condition is satisfied i.e., disjoint parts are independent. Other choices of q will correspond to a sub or super exponential relationship between the number of distinguishable states and distinguishable sub-systems, as found in strongly correlated systems. So inherently, the choice of q actually chooses the kind of correlations that the partitions Pi have, and it is with that caveat that we endeavor to establish the relationship between diversity of the parts to thewhole. 1.1. Purpose of current study In terms of the current study, we consider a general probability distributionwith a randomvariableX as shown in table 1 (signifying different types or categories), where xi denotes the i−th type, with probability pi and frequency fi.We ask the following question: ifPi denotes a partition of the indices {1,K,K} and D q Pi represents the diversity of that partition, what is the relationship between D q Pi and the diversity of allK types given by D q K ? And in particular, howdoes D q K change if the probability of a single type pK (say theK-th type) varies from0 to 1. In otherwords, we are interested in the functional relationship of D p q K K ( ) as pK varies from0 to 1. These results also address a key aspect of diversity and its relationshipwith the probability (or relative frequency) of the free type; namely, they show that as pK increases, there exists an optimal level of frequency pK ˆ so that themaximal diversity, which is equal to onemore than the diversity of the known types i.e., + D 1 q K 1 , is attained, and the diversity starts to decrease for all probabilities < < p p 1 K K ˆ . Furthermore, there exists a frequency pK ̃ so that = D p D q K K q K 1 ( ̃ ) . In other words, when the probability of theK-th type equals pK ̃ , the diversity of allK types is equal to the diversity of theK−1 types. And, in some sense, at = p p K K ̃ , it is as though theK-th type does not appear to exist. Also, for all probabilities < < p p 1 K K ̃ the diversity starts decreasing and when pK=1, theK-th type dominates entirely leading to a total diversity of 1 i.e. =  D p lim 1 p q K 1 K ( ) . We have organized the paper as follows. In section 2, we give a brief introduction to the idea ofmeasurement of diversity. In section 3, we prove the twomain theorems in the paper for DK 1 and 1 D q , 1 q K respectively. In section 4, we use the results from section 3 to investigate the function D p q K K ( ). Andwe finally concludewith a summary in section 5. At the outset, wemention that superscripts on the left are labels for diversities distinguishing the choices of q amongstHill numbers.We attach an appendix to clarify some of the notation used, in addition to explaining it in the paper as well. 2. A formal introduction to diversity Diversity, as ameasure, counts the ‘richness’ (number of types) of a distribution in relation to the ‘evenness’ (equal probability of occurrence) of its diversity types (MacArthur 1965,Hill 1973, Peet 1974, Jost 2006, Rajaram andCastellani 2012, 2014). The intuition behind this definition is that if all of the types in the distribution occur Table 1.General datasetwith complexity types xi eachhaving a probabilitypi and a frequency fi. X P F x1 p1 f1 x2 p2 f2 x3 p3 f3 M M M xJ pJ fJ M M M xK pK fK 2 J. Phys. Commun. 4 (2020) 045008 RRajaram andBCastellani