<figure><img src="image_1.png" /></figure>

HAL
open science

# Enlightening the Relationship Between Distribution and Regression Fitting

Hang Yu, Qian Yin, Ping Guo

► To cite this version:

Hang Yu, Qian Yin, Ping Guo. Enlightening the Relationship Between Distribution and Regression Fitting. 2nd International Conference on Intelligence Science (ICIS), Oct 2017, Shanghai, China. pp.222-227, 10.1007/978-3-319-68121-4_23. hal-01820940

HAL Id: hal-01820940

https://inria.hal.science/hal-01820940

Submitted on 22 Jun 2018

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Enlightening the Relationship between
Distribution and Regression Fitting

Hang Yu, Qian Yin(✉), and Ping Guo

Image Processing and Pattern Recognition Laboratory, Beijing Normal University,
Beijing 100875, China

yuhang@mail.bnu.edu.cn

yinqian@bnu.edu.cn

pguo@ieee.org

**Abstract.** Statistical distribution fitting and regression fitting are both classic methods to model data. There are slight connections and differences between them, as a result they outperform each other in different cases. A analysis model for processing natural data, say astronomical pulsar data in this paper, is proposed to improve data fitting method performance. Then the insight behind the comprehensive fitting model is given and discussed.

**Keywords:** Statistical Distribution, Regression, Model Selection, Goodness of Fit

## 1 Introduction

Statistics is a subject aiming to model the world by rather collecting and analyzing data than to inference and prove precisely. Lots of natural data have been fitted into statistical distributions with significant performances. And with these statistical distributions scientists can utilize their off-the-shelf distribution properties. While unfortunately the diversity of our world is far more complex than the level those statistical distributions can totally model. Alternatively, polynomial regression is also an approach to automatically find empirical laws from data[1]. It is proposed to fit various patterns of data, and has already shown its powerful fitting ability. Researchers who use polynomial regression are conveniently not required to preliminarily estimate data distribution[2]. Regrettably, by implementing polynomial regression we eventually get a polynomial expression with less prior properties than traditional statistical distributions have.

Given that features of natural scientific data are relatively fixed, and they will still be studied in the future, it is significant to determine the best fitting method for certain scientific data feature. And we creatively name it fitting label. As a solution to the dilemma, we propose a discriminative fitting model to help scientists automatically fitting scientific data into the optimal expression. Once scientists get the fitting label of a certain feature of natural scientific data, those later scientists are able to directly update the fitting expression parameters on
new data's arrival. We operated experiments on an astronomical data set, ANTF Pulsar Catalogue[3]. Besides, given the reliability of P value is still controversial[4], and mean squared error and KL divergence are trustworthy measures for both fields of statistics and machine learning, we complementally took them in consideration.

The structure of this paper is shown as follows: the comprehensive fitting model is proposed in section 2; and with it its corresponding experiments are showed and discussed in section 3 and 4; the section of conclusion comes as the end.

# 2 Methodology

## 2.1 Comprehensive Fitting Model

As is shown in Algorithm 1, the selection model consists of mainly 3 steps. Step 1 is to determine whether the data is subject to a certain natural distribution or not through significance testing. Step 2, fitting data into significant distributions and operating regression, then calculating their respective similarities. Step 3 is to classify the data's fitting label with distance-based classifier according to the similarities. Consequently, scientists are able to learn as more properties as possible of the data.

<caption>**Algorithm 1 Comprehensive Model**</caption>

**Input:** Scientific Data $\mathcal{D}$; 
Candidate Statistical Distribution Families $\Pi = \{\pi_1, ..., \pi_i, ..., \pi_m\}$; 
Selected Significance Testing Methods $\Gamma = \{\Gamma_1, ..., \Gamma_j, ...\Gamma_n\}$; 
Selected Similarity Measures $\Sigma = \{\sigma_1, ..., \sigma_j, ...\sigma_n\}$; 
Polynomial Regression Order $e$; 
Cross-Validation Times $t$ and Fold $k$.

**Output:** Fitting Labels $\pi^*$.

1: $\tilde{\Pi} \leftarrow Algorithm2(\mathcal{D}, \Pi, \Gamma, \Omega)$

2: $\hat{\Sigma} \leftarrow Algorithm3(\mathcal{D}, \Sigma, \tilde{\Pi})$

3: Fitting Labels $\pi^* \leftarrow kNN(\hat{\Sigma})$

return $\pi^*$

## 2.2 Statistical Significance Testing

As is shown in Algorithm 2, firstly we need human experts, astronomers for example, to give us candidate statistical distribution families for each feature. These distribution families with empirical parameters are then be tested one by one for statistical significance under selected statistical significance measures, usually P value. Concretely with specified natural science feature data, we perform statistical significance test to screen out distribution families that have
statistically large significance levels for the feature. Candidate statistical families and selected significance measures are given into the testing process. If the output P value reaches the threshold, a significance level of 0.05 by convention, the distribution will be recorded as a significant distribution.

**Algorithm 2 Statistical Significance Testing**

**Input:** Scientific Data $\mathcal{D}$;
Candidate Statistical Distribution Families $\Pi = \{\pi_1, ..., \pi_m\}$;
Selected Significance Testing Methods $\Gamma = \{\Gamma_1, ..., \Gamma_n\}$;
Testing Methods Weights $\Omega = \{\omega_1, ..., \omega_n\}$;
Significance Level $\tau$.

**Output:** Significant Statistical Distribution Families $\hat{\Pi} = \{\hat{\pi}_1, ..., \hat{\pi}_l\}$.

1: **for** each $\pi_i \in \Pi$ **do**

2: &nbsp;&nbsp;&nbsp;&nbsp;**for** each $\Gamma_j \in \Gamma$ **do**

3: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$\hat{\psi}_j^i \leftarrow \Gamma_j(\mathcal{D}, \pi_i)$;

4: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$\hat{\Psi}^i \leftarrow \hat{\Psi}^i \cup \hat{\psi}_j^i$;

5: &nbsp;&nbsp;&nbsp;&nbsp;**end for**

6: &nbsp;&nbsp;&nbsp;&nbsp;$\psi^i \leftarrow \Omega^\top \times \hat{\Psi}^i$;

7: &nbsp;&nbsp;&nbsp;&nbsp;**if** $\psi^i \ge \tau$ **then**

8: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$\hat{\Pi} \leftarrow \hat{\Pi} \cup \pi_i$;

9: &nbsp;&nbsp;&nbsp;&nbsp;**end if**

10: **end for**
&nbsp;&nbsp;&nbsp;&nbsp;**return** $\hat{\Pi}$

## 2.3 Similarity Evaluation

For this step, we use similarity measures to detect how well the expression fits with the data, which is shown in Algorithm 3. Significant statistical distributions obtained from last step are introduced here, and with it selected similarity measures. We perform t times k-fold cross-validation test to acquire the average similarity between the distribution and scientific data.

## 2.4 Distance-based Classification

It is the final outcome, the fitting label for the feature data, producing step. We can take those similarity measures as distances, and run a distance-based classifier to eventually select the expression with minimal distance as the optimal fitting label. Here we bring a classic classifier k-Nearest Neighbours(abbreviated as kNN) with its parameter k chosen 1.

# 3 Experiments

A logical starting point is to test a distribution's normality for its popularity in natural world[5]. While for features whose scatter diagram is shown in Fig.1, they
<table><caption>**Algorithm 3 Similarity Evaluation**</caption><thead><tr><th></th><th></th></tr></thead><tbody><tr><td>Input:</td><td>Pulsar Feature Data D;<br>Significant Statistical Distribution Families &#x010D; = {π<sub>1</sub>, ..., π<sub>l</sub>};<br>Selected Similarity Measures Σ = {σ<sub>1</sub>, ..., σ<sub>n</sub>};<br>Similarity Measure Weights Θ = {θ<sub>1</sub>, ..., θ<sub>n</sub>};<br>Polynomial Regression Highest Order e;<br>Cross-Validation Times t and Fold k.</td></tr><tr><td>Output:</td><td>Similarity/Goodness of Fit &#x010D; = {&#x010D;<sup>1</sup>, ..., &#x010D;<sup>l+1</sup>}.<br>1: for each π<sub>i</sub> ∈ &#x010D; do<br>2: for each σ<sub>j</sub> ∈ Σ do<br>3: σ&#x0302;<sub>j</sub><sup>i</sup> ← CV (t, k, D, π<sub>i</sub>, σ<sub>j</sub>);<br>4: σ&#x0302;<sup>i</sup> ← σ&#x0302;<sup>i</sup> ∪ σ&#x0302;<sub>j</sub><sup>i</sup>;<br>5: end for<br>6: σ&#x0302;<sup>i</sup> ← Θ<sup>T</sup> × σ&#x0302;<sup>i</sup>;<br>7: &#x010D; ← &#x010D; ∪ σ&#x0302;<sup>i</sup><br>8: end for<br>9: for each σ<sub>j</sub> ∈ Σ do<br>10: σ&#x0302;<sub>j</sub><sup>0</sup> ← CV (t, k, D, Polyfit (e), σ<sub>j</sub>);<br>11: σ&#x0302;<sup>0</sup> ← σ&#x0302;<sup>0</sup> ∪ σ&#x0302;<sub>j</sub><sup>0</sup>;<br>12: end for<br>13: σ&#x0302;<sup>0</sup> ← Θ<sup>T</sup> × σ&#x0302;<sup>0</sup>;<br>14: &#x010D; ← &#x010D; ∪ σ&#x0302;<sup>0</sup><br>return &#x010D;</td></tr></tbody></table>

seem obviously subject to lognormal distribution [6] [7]. We took the logarithm of that data first, then tested the resulting data's normality, which is shown in Fig.2. The red curve represented statistical distribution fitting, while the green one for polynomial regression fitting. As we know, logarithm of scale parameter and shape parameter of lognormal distribution are respectively expectation and standard deviation of normal distribution.

Significance results are shown in Table 1. For those classic statistical testing methods, the values recorded are their P values. Results show that lognormal has the highest significance among expert-selecting distributions.

<table><caption>**Table 1.** Significance Testing of P0</caption><thead><tr><th>P0</th><th>Normal</th><th>Lognormal</th><th>Power-law</th></tr></thead><tbody><tr><td>KS test</td><td>4.21349250398968713e-61</td><td>0.487092039062517</td><td>0.0</td></tr><tr><td>Shaprio-Wilk test</td><td>0.0</td><td>0.155729278922081</td><td>-</td></tr><tr><td>Kurtosis test</td><td>9.7779914011373847e-154</td><td>0.10520924193801827</td><td>-</td></tr><tr><td>Skewness test</td><td>0.0</td><td>0.97680081829094689</td><td>-</td></tr></tbody></table>

Similarity evaluation results are shown in Table 2. The polynomial order is empirically chosen 2. Thus lognormal distribution has the lowest relative entropy and mean squared error. So it is the optimal fitting label for pulsar period feature.
<figure><img src="image_3.png" /><figcaption>Fig. 1. Original Plot</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>Fig. 2. Logarithm Plot</figcaption></figure>

The temporary parameters of its corresponding normal distribution are mean $-0.53455956344351729$ and standard deviation $0.99686283436845236$.

<table><caption>Table 2. Similarity Evaluation of P0</caption><thead><tr><th>P0</th><th>Lognormal</th><th>Polynomial</th></tr></thead><tbody><tr><td>KL divergence</td><td>0.0188633190407</td><td>0.565469835801</td></tr><tr><td>MSE</td><td>2.49154711009</td><td>115.375894222</td></tr></tbody></table>

# 4 Discussions

As we know, the insight is a thought on inductive bias of Occam's razor and the principle of maximum entropy. Occam's razor has been the default heuristic technique in natural science for hundreds of years. It can be interpreted as stating that the one among competing hypotheses with the fewest assumptions should be selected[8][9]. The preference for simplicity also comes to machine learning's avoiding overfitting, a low-order fitting curve is almost always better than a high-order one. On the contrary, the second law of thermodynamics states that the total entropy of an isolated system can only increase over time. It is seen as an axiom of statistical thermodynamics. Here comes the principle of maximum entropy which states that the probability distribution that best represents the current state of knowledge is the one with largest entropy, and it also emphasizes a natural correspondence between statistical mechanics and information theory[10]. Numbers of statistical distributions widely used by natural scientists to model scientific data can be derived under the principle of maximum entropy. For any certain mean $\mu$ and variance $\sigma^2$, the maximum entropy distribution $p(x|\mu, \sigma^2)$ is normal distribution $N(\mu, \sigma^2)$[11].
# 5 Conclusions

For scientists we propose a discriminative fitting model through which diverse scientific data will be fitted into the optimal expression. And with our model as much as possible sample information can be used in future study. For data that is subject to a common statistical distribution, they then will be fitted into that distribution with parameters tuned by numerical optimization methods; comparatively for those who are not we model them by polynomial regression. Among main fields of natural science are still features whose fitting label remains unfixed, so it will be a great work to build a fitting label database with our model for scientists.

**Acknowledgments.** The research work in this paper was supported by the grants from National Natural Science Foundation of China (61472043, 61375045) and the Joint Research Fund in Astronomy (U1531242) under cooperative agreement between the NSFC and CAS, Beijing Natural Science Foundation (4142030). Prof. Qian Yin is the author to whom all the correspondence should be addressed.

# References

1. Edwards J R, Parry M E. On the use of polynomial regression equations as an alternative to difference scores in organizational research[J]. Academy of Management Journal, 1993, 36(6): 1577-1613.

2. Theil H. A rank-invariant method of linear and polynomial regression analysis, 3; confidence regions for the parameters of polynomial regression equations[J]. Stichting Mathematisch Centrum. Statistische Afdeling, 1950 (SP 5a/50/R): 1-16.

3. Manchester R N, Hobbs G B, Teoh A, et al. The Australia telescope national facility pulsar catalogue[J]. The Astronomical Journal, 2005, 129(4): 1993.

4. Nuzzo R. Statistical errors[J]. Nature, 2014, 506(7487): 150.

5. Faucher-Giguere C A, Kaspi V M. Birth and evolution of isolated radio pulsars[J]. The Astrophysical Journal, 2006, 643(1): 332.

6. Lorimer D R, Faulkner A J, Lyne A G, et al. The Parkes Multibeam Pulsar SurveyCVI. Discovery and timing of 142 pulsars and a Galactic population analysis[J]. Monthly Notices of the Royal Astronomical Society, 2006, 372(2): 777-800.

7. Bates S D, Lorimer D R, Rane A, et al. PsrPopPy: an open-source package for pulsar population simulations[J]. Monthly Notices of the Royal Astronomical Society, 2014: stu157.

8. Gauch H G. Scientific method in practice[M]. Cambridge University Press, 2003.

9. Hoffmann R, Minkin V I, Carpenter B K. Ockham's razor and chemistry[J]. Bulletin de la Société chimique de France, 1996, 133(2): 117-130.

10. Jaynes E T. Information Theory and Statistical Mechanics[J]. Physical Review, 1957, 106(4):620-630.

11. Jaynes E T. Probability theory: The logic of science[M]. Cambridge university press, 2003.