# Discrete Hyperparameter Optimization Model Based on Skewed Distribution

Authors: Yuqi Li
Journal: Mathematical Problems in Engineering
 (2022)

---

## Abstract

As for the machine learning algorithm, one of the main factors restricting its further large-scale application is the value of hyperparameter. Therefore, researchers have done a lot of original numerical optimization algorithms to ensure the validity of hyperparameter selection. Based on previous studies, this study innovatively puts forward a model generated using skewed distribution (gamma distribution) as hyperparameter fitting and combines the Bayesian estimation method and Gauss hypergeometric function to propose a mathematically optimal solution for discrete hyperparameter selection. The results show that under strict mathematical conditions, the value of discrete hyperparameters can be given a reasonable expected value. This heuristic parameter adjustment method based on prior conditions can improve the accuracy of some traditional models in experiments and then improve the application value of models. At the same time, through the empirical study of relevant datasets, the effectiveness of the parameter adjustment strategy proposed in this study is further proved.


---

## Body

## 1. Introduction

The whole Pareto/NBD model was created by Abramowitz and Stegun. The practical problem corresponding to the initial modeling is to analyze the behavior of consumers, especially the repeated purchase of related products, so that managers can evaluate the purchasing intention of future consumers and better arrange production and marketing activities. For the whole model itself, the description of consumer behavior is reflected in the following two important parts [1]. First, it is about the probability distribution or the probability distribution density of the consumer’s survival over the entire product purchasing cycle; second, it is about the description of the mathematical expectation of consumers’ purchasing behavior randomly selected in a certain consumption period in the future.For the construction of the whole Pareto/NBD model, the most critical part is to get the exact mathematical analytical expression of conditional mathematical expectation through appropriate mathematical assumptions and strict mathematical reasoning [2–4]. This work is very difficult for many researchers who are familiar with the SMC model. An important objective of this study was to prove the accuracy of some key conclusions about the Pareto/NBD model through mathematically deriving relevant impor`tant intermediate results (especially probability distribution function and probability distribution density function) within the framework of Pareto/NBD model. At the same time, the analytical framework of this model is combined with the hyperparameter optimization problem of discrete SMBO type to achieve the accuracy of mathematical description of the whole model.First of all, we need to introduce a special function, Gauss hypergeometric function, and the mathematical expression of this function in the form of power series [5] is as follows:(1)F12a,b;c;z=∑j=0∞ajbjcjzjj!,where c≠0,−1,−2,…, aj is the factorial power (Pochhammer’s symbol) operation, and specific mathematical calculation formula is aa+1⋯a+j−1. In contrast to the forming operation, it is about the parameter a of the ascending factorial power operation, and according to the assessment method of Abel, the convergence of the factorial power operation depends on the parameter z, in the case of z<1throughout the series converges to 1, in the case of z>1 tends to spread the whole series, z=1, the series converges to the c−a−b>0.The ascending joint operationaj can be expressed by two gamma functions with related parameters, which can be expressed by mathematical formula as follows:(2)aj=Γa+jΓa.With the help of this nature, the mathematical expression of the whole Gauss hyperparametric function can be further expressed as follows:(3)F12a,b;c;z=ΓcΓaΓb∑j=0∞Γa+jΓb+jΓc+jzjj!.According to the above mathematical expression form of Gauss hypergeometric function, combined with the nature of gamma function, the whole function is still expressed in discrete form, and the whole Gauss hypergeometric function is mutually symmetric for two parametersa and b [6], which is expressed in mathematical form as follows:(4)F12a,b;c;z=F12b,a;c;z.Furthermore, to further deepen the understanding of Gauss hypergeometric functions and facilitate the next model derivation, we must introduce the representation of Gauss hypergeometric functions in the form of Euler integral:(5)F12a,b;c;z=1Bb,c−b∫01tb−11−tc−b−11−zt−adt,c>b.Through the above mathematical expressions, we get the continuous expression of the Gauss hypergeometric function, whereB⋅,⋅ is the beta function.


## 2. Model Assumptions

Many aspects of BG/NBD model are similar to the Pareto/NBD model, but there are still many differences between models. The most important part is whether the hyperparameters change with each experiment/test. In the Pareto model, the hyperparameters change in any interval of each experiment/test interval, which is further independent of the selection and change in datasets. In the BG/NBD model, we assume that the timing of determining whether the hyperparameters change is controlled after each new dataset test; that is, the continuous discrimination interval in Pareto/NBD is changed into discrete decision interval, and then, the beta-geometric (BG) model can be established [7]. On the whole, the following five basic assumptions are needed for the new model.(1)
When dataset testing further influences the change in hyperparameters, the change in hyperparameters is subject to the Poisson process with parameterλ as a whole; that is to say,(6)PΘdt=x|λ=λdtxexp−λdtx!,x=0,1,2,…, where dt=Dt−∅ represents the difference between different datasets and empty sets, which can be the complexity change between datasets or the size change in datasets. For Poisson process distribution, the exponential distribution with parameter λ is followed between every two dataset experiments, and the mathematical expression is as follows:(7)gdj−dj−1|λ=λexp−λdj−dj−1,dj>dj−1≥0, where g stands for probability density function.(2)
For parameterλ, we assume that it is subject to the gamma distribution of parameters randα, and the mathematical expression is as follows:(8)gλ|r,α=αrλr−1exp−λαΓr,λ>0, where randα are the known and determined constants, which are determined by exogenous factors and not affected by the model itself.(3)
After each experiment/test, the probabilityp of the hyperparameter θ will remain unchanged. In other words, after any experiment/test, a judgment should be made on whether the next hyperparameter θ will change further. Then, the probability of the hyperparameter θ remaining unchanged should follow the geometric distribution of the parameter p:(9)Punchange=p1−pj−1,j=1,2,3,…, where j is the number of experiments/tests.(4)
To ensure that there is an analytic solution to the posterior distribution involving hyperparameters when solving the Bayesian estimation, it is necessary to assume that the parameterp is also determined by the exogenously given probability distribution, namely,(10)gp|a,b=pa−11−pb−1Ba,b,0≤p≤1, where aandb are the known and determined constants, which are determined by exogenous factors and not affected by the model itself.(5)
To eliminate the interaction between parameters and simplify the complexity of the overall model analysis, we need to assume the independence of parametersλ and p of the two models; that is, with the progress of the experiment/test, λ and p are independent of each other and not affected by each other.


## 3. Model Evolution

### 3.1. Derivation of Likelihood Function

We may consider that for the hyperparameterθ, it has experienced x changes in the known experimental/test interval 0,dT. These changes may occur in x interval d1,d2,…,dx in the whole test interval, which can be represented by the number line graph as follows.To ensure that a relatively complete likelihood function of relevant parameters can be derived [8], we carry out the following technical operation and further assumption:(1)
The probability of hyperparameterθ changing between d1 in the first experiment/test area is a standard exponential distribution model, and the probability of event occurrence should be λexp−λd1.(2)
The probability that the second change in hyperparameterθ happens to be in the second experiment/test interval d2 is in the first experiment/test interval d1. The probability of hyperparameter θ still continues to change with the product of the probability of d1 and d2 changes within the range of test; that is to say, in the whole experiment/test interval, the probability of θ hyperparameter change still obeys the standard of exponential distribution, but needs to multiply by the probability that the hyperparameter θ will still change in the first test interval. The specific expression is as follows:(11)1−pλexp−λd2−d1.(3)
Further extend to the probability that the hyperparameterθ changes at the xth experiment/test interval dx should be equal to the standard exponential distribution times the probability that the change will occur after the previous experiment/test interval dx−1, and the mathematical expression is as follows:(12)1−pλexp−λdx−dx−1.(4)
Finally, to ensure the likelihood function expression of completeness, we must make mandatory assumptions about the predicted change probability after the first experiment/testd1, assuming that the first test/test must be changed, but the changes can take 0 concrete or other step length, so overall hyperparameter θ in dx,dT interval constant probability is composed of two parts. The first part is the first time the experiment/test has changed, both of which do not change afterwards, and the second part is just the possibility just there is no change in dx,dT interval, where the specific expression is as follows:(13)p+1−pexp−λdT−dx.Further, we can obtain the likelihood function of the parameterd1,d2,…,dx,dT when λandp are known:(14)Lλ,p|d1,d2,…,dx,dT=λexp−λd11−pλexp−λd2−d1…1−pλexp−λdx−dx−1⋅p+1−pexp−λdT−dx=p1−px−1λxexp−λdx+1−pxλxexp−λdT.Just like our Pareto/NBD model above, the change in hyperparameterθ is still uncertain and random, and the prediction of the overall hyperparameter change depends on the historical information of the whole experiment/test: Θ=x,dx,dT.At the zero point of the initial experiment/test preparation stage, we must ensure that the hyperparameterθ will change, but there is a difference in the change step size, and then, we can get the probability likelihood function of the hyperparameter θ remaining at 0 in the whole historical information range of 0,dT, which can be further obtained using the exponential distribution:(15)Lλ|Θ=0,dT=exp−λdT.Further, for a single hyperparameterθ, we can obtain a probability likelihood function, and the mathematical expression is as follows:(16)Lλ,p|Θ=x,dT=1−pxλxexp−λdT+Ix>0p1−px−1λxexp−λdx,where Ix>0=1, when x>0, and x≤0 is less than minus, and the value of the whole is 0.


### 3.2. Derivation of the Probability Distribution of the Hyperparameterθ

Forθ hyperparameter probability distribution, the application of mathematical formula is expressed as PΘdt=x, where the Θdt represents the specific value of hyperparameter θ in the whole experiment/test interval dt, so Θdt becomes a random variable, so that you can establish the basic relationships between experiment/test interval and hyperparameter θ: Θdt≥x⇔Tx≤dt. Among them, Tx shows experiment/test range when θ hyperparameter is x changed. According to the corresponding relationship, we can get approximate mathematical expressions about PΘdt=x:(17)PΘdt=x=Pchange⋅PTx≤dt,Tx+1>dt+Ix>0⋅Punchange⋅PTx≤dt.Considering that the experimental/test interval in which the hyperparameterθ changes follows an exponential distribution (for specific content, see model hypothesis part), according to random variable for the hyperparameter itself, an important part of the probability distribution function is PTx≤dt,Tx+1>dt, a Poisson probability problem; among them, Θdt=x and PTx≤dt (Erlang—x) are the arrival time distribution. Furthermore, we can derive the conditional distribution function of hyperparameter with θ machine variable under the known condition of RP [9]:(18)PΘdt=x|λ,p=1−pxλdtxexp−λdtx!+Ix>0p1−px−1⋅1−exp−λdt∑j=0x−1λdtjj!.


### 3.3. Derivation of Mathematical Expectations for the Hyperparameterθ

Since the change in the hyperparameterθ follows the Poisson process with the parameter λdt, the mathematical expectation of the value change in the hyperparameter θ, i.e., EΘdt, should be λdt, within the given experimental/test interval. Then, for the hyperparameter θ, there are no longer changes in the experimental/test interval dτ≤dt; then, the mathematical expectation of EΘdt in 0,dτ with respect to the hyperparameter θ should be equal to λdτ.Then, for the change occurring outside the experimental/test intervaldτ, the probability distribution of the change in the value of the hyperparameter θ should follow the conditional probability model about λandp, that is,(19)Pdτ>dt=Pchange|λ,p=∑j=0∞1−pj−1exp−λdtλdtjj!=exp−λpdt.This expression further indicates the probability density function for no-answer experiment/test intervaldτ; that is to say, gdτ|λ,p=λpexp−λpdτ. It shall be highlighted that no-answer experiment/test interval dτ is an exponential form, but the relevant parameter arrangement depends on the probability for the recognition of each experiment/test p. However, in a Pareto/NBD model, the overall model assumes that no-answer experiment/test interval dτ is specified by the probability density function, and there is no relevant information for the discrimination of probability p. Then, we can further work out the conditional expectation expression about Θdt in the case that λandp know, and the specific mathematical expression is as follows:(20)EΘdt|λ,p=λdt⋅Pdτ>dt+∫0dtλdτgdτ|λ,pddτ.By substituting relevant intermediate variables into the above formula and further simplifying, we can get a very concise formula:(21)=1p−1pexp−λpdt.


### 3.4. Derivation of the Transcendental Posterior Expectation

By the content of the previous section, we discuss the core of the problem is to deduce the mathematical expectation ofEΘdt|λ,p involving θ hyperparameter under the condition of known parameters λandp, that is, under the premise of arrival rate λ of the test data and the know p, the conditional expectation expression of the hyperparameter θ is obtained, but in fact, according to the previous hypothesis, according to λandp, we set and select the two parameters that obey the distribution types and determined parameters. This part mainly focuses on the content of the derivation in λ to gamma distribution and p to beta distribution of cases, that is, to derive the posterior conditional expectation of Θdt=x involving the parameters r,α,a,b when we assume that the conditions (1) and (2) meet. The whole derivation process is relatively complex, and we will treat the proof content as an appendix. This summary mainly lists the main results in the derivation process.(1)
According to the content of Section 3.1, we have derived the probability likelihood function of the related parametersλandp when the hyperparameter and the whole experimental/test interval are known, in the form of formula (3). Furthermore, we will make parameters r,α obey the parameters for the gamma distribution and a,b, respectively, the assumptions for the beta distribution into the middle of the derived formula; note that we need to emphasize that the entire r,α,a,b parameter set is given and then get in the whole experiment/test cycle Θ=x,dx,dT known cases, and related parameter Θ=x,dx,dT probabilistic likelihood function is expressed as follows:(22)Lr,α,a,b|Θ=x,dx,dT=Ba,b+xBa,bΓr+xαrΓrα+dTr+x+Ix>0Ba+1,b+x−1Ba,bΓr+xαrΓrα+dxr+xAll the four variablesr,α,a,b in the above formula can be estimated by means of maximum-likelihood estimation. That is to say, in the whole known experiment/test cycle 0,dT, for Θ=x, the probability likelihood function after the log operation occurred in the dx experiment/test interval can be expressed as follows:(23)LLr,α,a,b=lnLr,α,a,b|Θ=x,dx,dT.The estimation of this formula can follow the general numerical optimization method and determine the optimal solution by finding the stagnation point of the first derivative and solving the extreme value, because the nature of the likelihood function must be convex.(2)
According to the conditional probability distribution of hyperparameterθ, the form is shown in formula (4). Considering that the specific distribution of λ,p has been given and the parameters have been determined, we can further express formula (4) as the conditional probability distribution under the condition that parameter set r,α,a,b is known:(24)PΘdt=x|r,α,a,b=Ba,b+xBa,bΓr+xΓrx!αα+dtrdtα+dtx+Ix>0Ba+1,b+x−1Ba,b⋅1−αα+dtr∑j=0x−1Γr+jΓrj!dtα+dtj.(3)
Finally, according to the condition of known parameters ofλ,p, the hyperparameter θ conditional expectation, the concrete mathematical expression such as formula (5), combined with the λ,p two parameters, follows gamma distribution and beta distribution in concrete form, and we can get in any experiment/test interval length dt, under the condition of the parameters in the parameter set r,α,a,b known, hyperparameter θ conditional expectation, which can also be called the posterior conditional expectation; the mathematic expression of the specific is as follows:(25)EΘdt|r,α,a,b=a+b−1a−11−αα+dt2rF1r,b;a+b−1;dtα+dt, where F1· is Gauss hypergeometric function; refer to the explanation in the proof for the simple derivation, and refer to the relevant introduction in the model introduction for the specific content.It is important to note that the final expression of posterior conditional expectation about hyperparameterθ needs a value about Gauss hypergeometric function; of course, this value has nothing to do with hyperparameter θ. On the contrary, due to the parameter set r,α,a,b can be obtained by maximum-likelihood estimate of the traditional method, and the experiment/test interval length is also a known quantity, so the overall Gauss hypergeometric function value is also determined. Compared with other estimated results, using the Gauss hypergeometric function to express the hyperparameter θ of the posterior conditional expectation is more direct and simple, and polynomial sequence can be used for approximate calculation, so the convenience for the late of simulation is very good, which can greatly simplify the computing complexity and try to solve the corresponding parameter selection in a timely manner.Of course, the above content is just based on known sequence informationΘ=x,dx,dT cases and makes the probability model for how hyperparameter θ values. In addition to accommodating more known conditions and historical information as much as possible to make accurate simulation of the structure of past data, the final purpose of establishing BP/NBD model shall also focus on a given new experiment/test dataset case and how to utilize the known datasets and the relationship between the parameter θ, given the new dataset the parameters of an approximate estimate [10], and the key of the whole model application on the new dataset corresponds to the hyperparameter of the prediction is that we more concern.Θ=x,dx,dT in the history of the experiment/test information has been confirmed, we can have θ hyperparameter values on the new experiment/test interval dt, we give an approximate posterior conditional expectation, the specific detailed process can see behind a certificate, and here we give the concrete mathematical expression of this posterior conditional expectation:(26)Eϒdt|Θ=x,dx,dT,r,α,a,b=a+b+x−1/a−11−α+dT/α+dT+dt2rF1r+x,b+x;a+b+x−1;dt/α+dT+dt1+Ix>0a/b+x−1α+dT/α+dxr+x,where Θ=x,dx,dT stands for the known experimental/test information, r,α,a,b stands for the known parameter set by calculation, and F1· stands for the Gauss hypergeometric function.Once again, the whole formula uses the Gauss hypergeometric function, and a mathematical expression formula is the same, of course, where the Gauss hypergeometric function parameters are known, and the calculation of the precise value is guaranteed. To reduce the computation burden on the computer simulation process, the actual operation often uses polynomial equations to approximate its numerical calculation and will not cause enormous computation burden. The rest of the whole formulas are simple numerical calculations and would not form the computing burden.The overall content of the appendix mainly focuses on the mathematical description of the key process in the model derivation process. On the one hand, through mathematical rigour, it indicates the credibility of model content; on the other hand, it also provides mathematical support for transforming the mathematical model into computer model. The main purpose is the mathematical expectation for hyperparameterθ, EΘdt, and ultimately predictable θ hyperparameter posterior mathematical expectation of mathematical deduction, and the derivation process of the intermediate links is the Euler integral for Gauss hypergeometric function:(27)F12a,b;c;z=1Bb,c−b∫01tb−11−tc−b−11−zt−adt,c>b.


### 3.5. Derivation of Mathematical ExpectationEΘdt for the Hyperparameter θ

To get the mathematical expectationEΘdt about hyperparameter θ, according to the condition of known parameter set λ,p, we must use formula about the conditional expectation of hyperparameter θ and assumptions about the distribution of λ and p, λ, parameters of r,α gamma distribution on p obedience of a,b beta distribution parameters; first of all, we will bring the probability distribution density of gλ|r,α=αrλr−1exp−λα/Γr,λ>0, related to λ into formula (4) and get preliminary mathematical expressions:(28)EΘdt|r,α,p=1p−αrpα+pdtr.Further, we substitute the beta probability distribution densitygp|a,b=pa−11−pb−1/Ba,b,0≤p≤1, of p subject to the overall conditional expectation formula of hyperparameter θ, and we first calculate a definite integral as follows:(29)∫011ppa−11−pb−1Ba,bdp=a+b−1a−1.Further, a more complex definite integral needs to be calculated:(30)∫01αrpα+pdtrpa−11−pb−1Ba,bdp=αr1Ba,b∫01pa−21−pb−1α+pdt−rdp.To ensure that the definite integral form at the end is consistent with the form of Gauss hypergeometric function, we need to make a variable substitution, soq=1−p; of course, in the process of variable substitution, the form of the differential will also change dp=−dq. The integral form above further becomes the following:(31)=αα+dtr1Ba,b∫01qb−11−qa−21−q⋅dtα+dt−rdq.The Euler integral form and parameterr,b;a+b−1;dt/α+dt of the Gauss hypergeometric function mentioned above are further applied, and the corresponding parameters in the integral form are expressed, respectively; then, the definite integral can be further formalized as follows:(32)=αα+dtrBa−1,bBa,b2F1r,b;a+b−1;dtα+dt,where F1· is a Gauss hypergeometric function.Finally, the conditional expectation formula of the hyperparameterθ can be obtained, and its mathematical expression is as follows:(33)EΘdt|r,α,a,b=a+b−1a−11−αα+dtrF12r,b;a+b−1;dtα+dt.


### 3.6. Derivation of the Conditional ExpectationEϒdt|Θ=x,dx,dT for the Hyperparameter θ with Known Historical Information

We further define the variation ofdT,dT+t hyperparameter θ in the new experimental/test interval of ϒdt. As a prediction of hyperparameter θ in the new dataset, we focus on the conditional expectation of the value of hyperparameter B under the premise that the historical information is known; i.e., Θ=x,dx,dT is determined, which is expressed as Eϒdt|Θ=x,dx,dT by mathematical formula.If the hyperparameterθ still changes further in dT experiments/tests, then the conditional mathematical expectation of ϒdt, the corresponding random variable, can be expressed as follows:(34)Eϒdt|λ,p=1p−1pexp−λpdt.Then, further, we need to make sure that in the new experiment/test sequence, how much probability that hyperparameterθ will change. According to our hypothesis in the second section, all the experiment/test sequence for each group in the initial state can keep a change state, the purpose of which is to ensure the overall model can be used to measure is in complete probability space, that is to say, in dT/tests, hyperparameter θ keeps changing the state of the conditional probability, which can be expressed as follows:(35)Pchange|Θ=0,dT,λ,p=1.As for0,dT experiment/test interval, the conditional probability of te hyperparameter θ remaining changing state can be approximately expressed, in the case that historical information Θ=x,dx,dT and parameters λ and p have been determined:(36)Pchange|Θ=x,dx,dT,λ,p=1−pexp−λdT−dxp+1−pexp−λdT−dx.Among them, the molecules represented indx/tests still keep change state, but that in the experiment/test interval of dx,dT keep zero step change probability, and the denominator is the probability that nothing has changed from the beginning of the experiment/test to the end of the historical information, as the case is in assumption (4) in Section 3.1.Here, we apply a small trick of conditional probability. For the clever treatment of “1,”1−pexp−λdT−dx/1−pexp−λdT−dx=1; multiplied by the above equation, we can get an important intermediate result:(37)Pchange|Θ=x,dx,dT,λ,p=1−pxλxexp−λdTLλ,p|Θ=x,dx,dT,where Lλ,p|Θ=x,dT=1−pxλxexp−λdT+Ix>0p1−px−1λxexp−λdx, as previously concluded in formula (3); note that we need to assume that, at x=0, the whole A2 formula should be equal to 1.We multiply the two important intermediate variables A1 and A2 to obtain the following further results:(38)Eϒdt|Θ=x,dx,dT,λ,p=1−pxλxexp−λdT1/p−1/pexp−λdt/pLλ,p|Θ=x,dx,dT=p−11−pxλxexp−λdT−p−11−pxλxexp−λdT+pdtLλ,p|Θ=x,dx,dT.It should be noted that the case of A1 whenx=0 is eliminated, because the premise for the existence of the entire posterior probability and the posterior mathematical expectation is that the entire historical information should remain in the new interval dT,dT+t state at dT, and the hyperparameter θ must be able to connect with each other.Because the above results are involved parametersλ and p, they are still incomplete expressions of random variable ϒdt on posterior mathematical expectation. Therefore, according to Section 2 of model assumptions, gλ|r,α=αrλr−1exp−λα/Γr,λ>0, and gp|a,b=pa−11−pb−1/Ba,b,0≤p≤1; further processing, since the probability distribution density of these two parameters is continuous, we can adopt the method of integral and further improve A3 posterior mathematical expectation and add more uncertainty information, especially the hidden information about parameters λ,p that express explicit use of parameter set r,α,a,b:(39)Eϒdt|Θ=x,dx,dT,r,α,a,b=∫01∫0∞Eϒdt|Θ=x,dx,dT,λ,p⋅gλ,p|Θ=x,dx,dT,r,α,a,bdλdp,where gλ,p|Θ=x,dx,dT,r,α,a,b is the joint posterior probability distribution density of the parameters λ,p.According to the basic theory of Bayesian estimation and Bayesian optimization, under the condition that the parametersλ,p is independent, their joint posterior probability distribution density can be expressed in the following complex form:(40)gλ,p|Θ=x,dx,dT,r,α,a,b=Lλ,p|Θ=x,dx,dTgλ|r,αgp|a,bLr,α,a,b|Θ=x,dx,dT.By combining the key intermediate variables A5 and A3 in formula A4, the posterior mathematical expectation related to the hyperparameterθ can be further deduced:(41)Eϒdt|Θ=x,dx,dT,r,α,a,b=A−BLr,α,a,b|Θ=x,dx,dT,where two parameters A and B, respectively, represent the following two operation results:ForA,(42)A=∫01∫0∞p−11−pxλxexp−λdTgλ|r,αgp|a,bdλdp=Ba−1,b+xBa,bΓr+xαrΓrα+dTr+x.ForB,(43)B=∫01∫0∞p−11−pxλxexp−λdT+pdtgλ|r,αgp|a,bdλdp=∫01Pa−21−pb+x−1Ba,b∫0∞αrλr+x−1exp−λdT+pdt+αΓrdλdp=Γr+xαrΓrBa,b∫01pa−21−pb+x−1α+dT+pdt−r+xdp.To achieve formal matching with the Gauss hypergeometric function in the form of Euler integral, we need to further realize variable substitution, and letq=1−p and also make corresponding change dp=−dq for the differential variable. Further calculation results are as follows:(44)=Γr+xαrΓrBa,bα+dT+dtr+x⋅∫011−qa−2qb+x−11−dtqα+dT+dt−r+xdq.By combining the Gauss hypergeometric function in the form of Euler integral, we get the parameterr,b;a+b−1;dt/α+dt, which corresponds to the corresponding parameter expression in the form of integral function respectively, and we get:(45)=Ba−1,b+xBa,bΓr+xαrΓrα+dT+dtr+x⋅2F1r+x,b+x;a+b+x−1;dtα+dT+dt.Under the condition of formula A6, according to the mathematical expression formula ofLr,α,a,b|Θ=x,dx,dT (6), the middle of the two key variables A7 and A8, after reduction, we can get the ideal result, under the BP/NBD model; the mathematical expectation of overall θ hyperparameters under the posteriori probability has analytical solution, and the mathematic expression of the specific is as follows:(46)Eϒdt|Θ=x,dx,dT,r,α,a,b=a+b+x−1/a−11−α+dT/α+dT+dtrF12r+x,b+x;a+b+x−1;dt/α+dT+dt1+Ix>0a/b+x−1α+dT/α+dxr+x.


## 3.1. Derivation of Likelihood Function

We may consider that for the hyperparameterθ, it has experienced x changes in the known experimental/test interval 0,dT. These changes may occur in x interval d1,d2,…,dx in the whole test interval, which can be represented by the number line graph as follows.To ensure that a relatively complete likelihood function of relevant parameters can be derived [8], we carry out the following technical operation and further assumption:(1)
The probability of hyperparameterθ changing between d1 in the first experiment/test area is a standard exponential distribution model, and the probability of event occurrence should be λexp−λd1.(2)
The probability that the second change in hyperparameterθ happens to be in the second experiment/test interval d2 is in the first experiment/test interval d1. The probability of hyperparameter θ still continues to change with the product of the probability of d1 and d2 changes within the range of test; that is to say, in the whole experiment/test interval, the probability of θ hyperparameter change still obeys the standard of exponential distribution, but needs to multiply by the probability that the hyperparameter θ will still change in the first test interval. The specific expression is as follows:(11)1−pλexp−λd2−d1.(3)
Further extend to the probability that the hyperparameterθ changes at the xth experiment/test interval dx should be equal to the standard exponential distribution times the probability that the change will occur after the previous experiment/test interval dx−1, and the mathematical expression is as follows:(12)1−pλexp−λdx−dx−1.(4)
Finally, to ensure the likelihood function expression of completeness, we must make mandatory assumptions about the predicted change probability after the first experiment/testd1, assuming that the first test/test must be changed, but the changes can take 0 concrete or other step length, so overall hyperparameter θ in dx,dT interval constant probability is composed of two parts. The first part is the first time the experiment/test has changed, both of which do not change afterwards, and the second part is just the possibility just there is no change in dx,dT interval, where the specific expression is as follows:(13)p+1−pexp−λdT−dx.Further, we can obtain the likelihood function of the parameterd1,d2,…,dx,dT when λandp are known:(14)Lλ,p|d1,d2,…,dx,dT=λexp−λd11−pλexp−λd2−d1…1−pλexp−λdx−dx−1⋅p+1−pexp−λdT−dx=p1−px−1λxexp−λdx+1−pxλxexp−λdT.Just like our Pareto/NBD model above, the change in hyperparameterθ is still uncertain and random, and the prediction of the overall hyperparameter change depends on the historical information of the whole experiment/test: Θ=x,dx,dT.At the zero point of the initial experiment/test preparation stage, we must ensure that the hyperparameterθ will change, but there is a difference in the change step size, and then, we can get the probability likelihood function of the hyperparameter θ remaining at 0 in the whole historical information range of 0,dT, which can be further obtained using the exponential distribution:(15)Lλ|Θ=0,dT=exp−λdT.Further, for a single hyperparameterθ, we can obtain a probability likelihood function, and the mathematical expression is as follows:(16)Lλ,p|Θ=x,dT=1−pxλxexp−λdT+Ix>0p1−px−1λxexp−λdx,where Ix>0=1, when x>0, and x≤0 is less than minus, and the value of the whole is 0.


## 3.2. Derivation of the Probability Distribution of the Hyperparameterθ

Forθ hyperparameter probability distribution, the application of mathematical formula is expressed as PΘdt=x, where the Θdt represents the specific value of hyperparameter θ in the whole experiment/test interval dt, so Θdt becomes a random variable, so that you can establish the basic relationships between experiment/test interval and hyperparameter θ: Θdt≥x⇔Tx≤dt. Among them, Tx shows experiment/test range when θ hyperparameter is x changed. According to the corresponding relationship, we can get approximate mathematical expressions about PΘdt=x:(17)PΘdt=x=Pchange⋅PTx≤dt,Tx+1>dt+Ix>0⋅Punchange⋅PTx≤dt.Considering that the experimental/test interval in which the hyperparameterθ changes follows an exponential distribution (for specific content, see model hypothesis part), according to random variable for the hyperparameter itself, an important part of the probability distribution function is PTx≤dt,Tx+1>dt, a Poisson probability problem; among them, Θdt=x and PTx≤dt (Erlang—x) are the arrival time distribution. Furthermore, we can derive the conditional distribution function of hyperparameter with θ machine variable under the known condition of RP [9]:(18)PΘdt=x|λ,p=1−pxλdtxexp−λdtx!+Ix>0p1−px−1⋅1−exp−λdt∑j=0x−1λdtjj!.


## 3.3. Derivation of Mathematical Expectations for the Hyperparameterθ

Since the change in the hyperparameterθ follows the Poisson process with the parameter λdt, the mathematical expectation of the value change in the hyperparameter θ, i.e., EΘdt, should be λdt, within the given experimental/test interval. Then, for the hyperparameter θ, there are no longer changes in the experimental/test interval dτ≤dt; then, the mathematical expectation of EΘdt in 0,dτ with respect to the hyperparameter θ should be equal to λdτ.Then, for the change occurring outside the experimental/test intervaldτ, the probability distribution of the change in the value of the hyperparameter θ should follow the conditional probability model about λandp, that is,(19)Pdτ>dt=Pchange|λ,p=∑j=0∞1−pj−1exp−λdtλdtjj!=exp−λpdt.This expression further indicates the probability density function for no-answer experiment/test intervaldτ; that is to say, gdτ|λ,p=λpexp−λpdτ. It shall be highlighted that no-answer experiment/test interval dτ is an exponential form, but the relevant parameter arrangement depends on the probability for the recognition of each experiment/test p. However, in a Pareto/NBD model, the overall model assumes that no-answer experiment/test interval dτ is specified by the probability density function, and there is no relevant information for the discrimination of probability p. Then, we can further work out the conditional expectation expression about Θdt in the case that λandp know, and the specific mathematical expression is as follows:(20)EΘdt|λ,p=λdt⋅Pdτ>dt+∫0dtλdτgdτ|λ,pddτ.By substituting relevant intermediate variables into the above formula and further simplifying, we can get a very concise formula:(21)=1p−1pexp−λpdt.


## 3.4. Derivation of the Transcendental Posterior Expectation

By the content of the previous section, we discuss the core of the problem is to deduce the mathematical expectation ofEΘdt|λ,p involving θ hyperparameter under the condition of known parameters λandp, that is, under the premise of arrival rate λ of the test data and the know p, the conditional expectation expression of the hyperparameter θ is obtained, but in fact, according to the previous hypothesis, according to λandp, we set and select the two parameters that obey the distribution types and determined parameters. This part mainly focuses on the content of the derivation in λ to gamma distribution and p to beta distribution of cases, that is, to derive the posterior conditional expectation of Θdt=x involving the parameters r,α,a,b when we assume that the conditions (1) and (2) meet. The whole derivation process is relatively complex, and we will treat the proof content as an appendix. This summary mainly lists the main results in the derivation process.(1)
According to the content of Section 3.1, we have derived the probability likelihood function of the related parametersλandp when the hyperparameter and the whole experimental/test interval are known, in the form of formula (3). Furthermore, we will make parameters r,α obey the parameters for the gamma distribution and a,b, respectively, the assumptions for the beta distribution into the middle of the derived formula; note that we need to emphasize that the entire r,α,a,b parameter set is given and then get in the whole experiment/test cycle Θ=x,dx,dT known cases, and related parameter Θ=x,dx,dT probabilistic likelihood function is expressed as follows:(22)Lr,α,a,b|Θ=x,dx,dT=Ba,b+xBa,bΓr+xαrΓrα+dTr+x+Ix>0Ba+1,b+x−1Ba,bΓr+xαrΓrα+dxr+xAll the four variablesr,α,a,b in the above formula can be estimated by means of maximum-likelihood estimation. That is to say, in the whole known experiment/test cycle 0,dT, for Θ=x, the probability likelihood function after the log operation occurred in the dx experiment/test interval can be expressed as follows:(23)LLr,α,a,b=lnLr,α,a,b|Θ=x,dx,dT.The estimation of this formula can follow the general numerical optimization method and determine the optimal solution by finding the stagnation point of the first derivative and solving the extreme value, because the nature of the likelihood function must be convex.(2)
According to the conditional probability distribution of hyperparameterθ, the form is shown in formula (4). Considering that the specific distribution of λ,p has been given and the parameters have been determined, we can further express formula (4) as the conditional probability distribution under the condition that parameter set r,α,a,b is known:(24)PΘdt=x|r,α,a,b=Ba,b+xBa,bΓr+xΓrx!αα+dtrdtα+dtx+Ix>0Ba+1,b+x−1Ba,b⋅1−αα+dtr∑j=0x−1Γr+jΓrj!dtα+dtj.(3)
Finally, according to the condition of known parameters ofλ,p, the hyperparameter θ conditional expectation, the concrete mathematical expression such as formula (5), combined with the λ,p two parameters, follows gamma distribution and beta distribution in concrete form, and we can get in any experiment/test interval length dt, under the condition of the parameters in the parameter set r,α,a,b known, hyperparameter θ conditional expectation, which can also be called the posterior conditional expectation; the mathematic expression of the specific is as follows:(25)EΘdt|r,α,a,b=a+b−1a−11−αα+dt2rF1r,b;a+b−1;dtα+dt, where F1· is Gauss hypergeometric function; refer to the explanation in the proof for the simple derivation, and refer to the relevant introduction in the model introduction for the specific content.It is important to note that the final expression of posterior conditional expectation about hyperparameterθ needs a value about Gauss hypergeometric function; of course, this value has nothing to do with hyperparameter θ. On the contrary, due to the parameter set r,α,a,b can be obtained by maximum-likelihood estimate of the traditional method, and the experiment/test interval length is also a known quantity, so the overall Gauss hypergeometric function value is also determined. Compared with other estimated results, using the Gauss hypergeometric function to express the hyperparameter θ of the posterior conditional expectation is more direct and simple, and polynomial sequence can be used for approximate calculation, so the convenience for the late of simulation is very good, which can greatly simplify the computing complexity and try to solve the corresponding parameter selection in a timely manner.Of course, the above content is just based on known sequence informationΘ=x,dx,dT cases and makes the probability model for how hyperparameter θ values. In addition to accommodating more known conditions and historical information as much as possible to make accurate simulation of the structure of past data, the final purpose of establishing BP/NBD model shall also focus on a given new experiment/test dataset case and how to utilize the known datasets and the relationship between the parameter θ, given the new dataset the parameters of an approximate estimate [10], and the key of the whole model application on the new dataset corresponds to the hyperparameter of the prediction is that we more concern.Θ=x,dx,dT in the history of the experiment/test information has been confirmed, we can have θ hyperparameter values on the new experiment/test interval dt, we give an approximate posterior conditional expectation, the specific detailed process can see behind a certificate, and here we give the concrete mathematical expression of this posterior conditional expectation:(26)Eϒdt|Θ=x,dx,dT,r,α,a,b=a+b+x−1/a−11−α+dT/α+dT+dt2rF1r+x,b+x;a+b+x−1;dt/α+dT+dt1+Ix>0a/b+x−1α+dT/α+dxr+x,where Θ=x,dx,dT stands for the known experimental/test information, r,α,a,b stands for the known parameter set by calculation, and F1· stands for the Gauss hypergeometric function.Once again, the whole formula uses the Gauss hypergeometric function, and a mathematical expression formula is the same, of course, where the Gauss hypergeometric function parameters are known, and the calculation of the precise value is guaranteed. To reduce the computation burden on the computer simulation process, the actual operation often uses polynomial equations to approximate its numerical calculation and will not cause enormous computation burden. The rest of the whole formulas are simple numerical calculations and would not form the computing burden.The overall content of the appendix mainly focuses on the mathematical description of the key process in the model derivation process. On the one hand, through mathematical rigour, it indicates the credibility of model content; on the other hand, it also provides mathematical support for transforming the mathematical model into computer model. The main purpose is the mathematical expectation for hyperparameterθ, EΘdt, and ultimately predictable θ hyperparameter posterior mathematical expectation of mathematical deduction, and the derivation process of the intermediate links is the Euler integral for Gauss hypergeometric function:(27)F12a,b;c;z=1Bb,c−b∫01tb−11−tc−b−11−zt−adt,c>b.


## 3.5. Derivation of Mathematical ExpectationEΘdt for the Hyperparameter θ

To get the mathematical expectationEΘdt about hyperparameter θ, according to the condition of known parameter set λ,p, we must use formula about the conditional expectation of hyperparameter θ and assumptions about the distribution of λ and p, λ, parameters of r,α gamma distribution on p obedience of a,b beta distribution parameters; first of all, we will bring the probability distribution density of gλ|r,α=αrλr−1exp−λα/Γr,λ>0, related to λ into formula (4) and get preliminary mathematical expressions:(28)EΘdt|r,α,p=1p−αrpα+pdtr.Further, we substitute the beta probability distribution densitygp|a,b=pa−11−pb−1/Ba,b,0≤p≤1, of p subject to the overall conditional expectation formula of hyperparameter θ, and we first calculate a definite integral as follows:(29)∫011ppa−11−pb−1Ba,bdp=a+b−1a−1.Further, a more complex definite integral needs to be calculated:(30)∫01αrpα+pdtrpa−11−pb−1Ba,bdp=αr1Ba,b∫01pa−21−pb−1α+pdt−rdp.To ensure that the definite integral form at the end is consistent with the form of Gauss hypergeometric function, we need to make a variable substitution, soq=1−p; of course, in the process of variable substitution, the form of the differential will also change dp=−dq. The integral form above further becomes the following:(31)=αα+dtr1Ba,b∫01qb−11−qa−21−q⋅dtα+dt−rdq.The Euler integral form and parameterr,b;a+b−1;dt/α+dt of the Gauss hypergeometric function mentioned above are further applied, and the corresponding parameters in the integral form are expressed, respectively; then, the definite integral can be further formalized as follows:(32)=αα+dtrBa−1,bBa,b2F1r,b;a+b−1;dtα+dt,where F1· is a Gauss hypergeometric function.Finally, the conditional expectation formula of the hyperparameterθ can be obtained, and its mathematical expression is as follows:(33)EΘdt|r,α,a,b=a+b−1a−11−αα+dtrF12r,b;a+b−1;dtα+dt.


## 3.6. Derivation of the Conditional ExpectationEϒdt|Θ=x,dx,dT for the Hyperparameter θ with Known Historical Information

We further define the variation ofdT,dT+t hyperparameter θ in the new experimental/test interval of ϒdt. As a prediction of hyperparameter θ in the new dataset, we focus on the conditional expectation of the value of hyperparameter B under the premise that the historical information is known; i.e., Θ=x,dx,dT is determined, which is expressed as Eϒdt|Θ=x,dx,dT by mathematical formula.If the hyperparameterθ still changes further in dT experiments/tests, then the conditional mathematical expectation of ϒdt, the corresponding random variable, can be expressed as follows:(34)Eϒdt|λ,p=1p−1pexp−λpdt.Then, further, we need to make sure that in the new experiment/test sequence, how much probability that hyperparameterθ will change. According to our hypothesis in the second section, all the experiment/test sequence for each group in the initial state can keep a change state, the purpose of which is to ensure the overall model can be used to measure is in complete probability space, that is to say, in dT/tests, hyperparameter θ keeps changing the state of the conditional probability, which can be expressed as follows:(35)Pchange|Θ=0,dT,λ,p=1.As for0,dT experiment/test interval, the conditional probability of te hyperparameter θ remaining changing state can be approximately expressed, in the case that historical information Θ=x,dx,dT and parameters λ and p have been determined:(36)Pchange|Θ=x,dx,dT,λ,p=1−pexp−λdT−dxp+1−pexp−λdT−dx.Among them, the molecules represented indx/tests still keep change state, but that in the experiment/test interval of dx,dT keep zero step change probability, and the denominator is the probability that nothing has changed from the beginning of the experiment/test to the end of the historical information, as the case is in assumption (4) in Section 3.1.Here, we apply a small trick of conditional probability. For the clever treatment of “1,”1−pexp−λdT−dx/1−pexp−λdT−dx=1; multiplied by the above equation, we can get an important intermediate result:(37)Pchange|Θ=x,dx,dT,λ,p=1−pxλxexp−λdTLλ,p|Θ=x,dx,dT,where Lλ,p|Θ=x,dT=1−pxλxexp−λdT+Ix>0p1−px−1λxexp−λdx, as previously concluded in formula (3); note that we need to assume that, at x=0, the whole A2 formula should be equal to 1.We multiply the two important intermediate variables A1 and A2 to obtain the following further results:(38)Eϒdt|Θ=x,dx,dT,λ,p=1−pxλxexp−λdT1/p−1/pexp−λdt/pLλ,p|Θ=x,dx,dT=p−11−pxλxexp−λdT−p−11−pxλxexp−λdT+pdtLλ,p|Θ=x,dx,dT.It should be noted that the case of A1 whenx=0 is eliminated, because the premise for the existence of the entire posterior probability and the posterior mathematical expectation is that the entire historical information should remain in the new interval dT,dT+t state at dT, and the hyperparameter θ must be able to connect with each other.Because the above results are involved parametersλ and p, they are still incomplete expressions of random variable ϒdt on posterior mathematical expectation. Therefore, according to Section 2 of model assumptions, gλ|r,α=αrλr−1exp−λα/Γr,λ>0, and gp|a,b=pa−11−pb−1/Ba,b,0≤p≤1; further processing, since the probability distribution density of these two parameters is continuous, we can adopt the method of integral and further improve A3 posterior mathematical expectation and add more uncertainty information, especially the hidden information about parameters λ,p that express explicit use of parameter set r,α,a,b:(39)Eϒdt|Θ=x,dx,dT,r,α,a,b=∫01∫0∞Eϒdt|Θ=x,dx,dT,λ,p⋅gλ,p|Θ=x,dx,dT,r,α,a,bdλdp,where gλ,p|Θ=x,dx,dT,r,α,a,b is the joint posterior probability distribution density of the parameters λ,p.According to the basic theory of Bayesian estimation and Bayesian optimization, under the condition that the parametersλ,p is independent, their joint posterior probability distribution density can be expressed in the following complex form:(40)gλ,p|Θ=x,dx,dT,r,α,a,b=Lλ,p|Θ=x,dx,dTgλ|r,αgp|a,bLr,α,a,b|Θ=x,dx,dT.By combining the key intermediate variables A5 and A3 in formula A4, the posterior mathematical expectation related to the hyperparameterθ can be further deduced:(41)Eϒdt|Θ=x,dx,dT,r,α,a,b=A−BLr,α,a,b|Θ=x,dx,dT,where two parameters A and B, respectively, represent the following two operation results:ForA,(42)A=∫01∫0∞p−11−pxλxexp−λdTgλ|r,αgp|a,bdλdp=Ba−1,b+xBa,bΓr+xαrΓrα+dTr+x.ForB,(43)B=∫01∫0∞p−11−pxλxexp−λdT+pdtgλ|r,αgp|a,bdλdp=∫01Pa−21−pb+x−1Ba,b∫0∞αrλr+x−1exp−λdT+pdt+αΓrdλdp=Γr+xαrΓrBa,b∫01pa−21−pb+x−1α+dT+pdt−r+xdp.To achieve formal matching with the Gauss hypergeometric function in the form of Euler integral, we need to further realize variable substitution, and letq=1−p and also make corresponding change dp=−dq for the differential variable. Further calculation results are as follows:(44)=Γr+xαrΓrBa,bα+dT+dtr+x⋅∫011−qa−2qb+x−11−dtqα+dT+dt−r+xdq.By combining the Gauss hypergeometric function in the form of Euler integral, we get the parameterr,b;a+b−1;dt/α+dt, which corresponds to the corresponding parameter expression in the form of integral function respectively, and we get:(45)=Ba−1,b+xBa,bΓr+xαrΓrα+dT+dtr+x⋅2F1r+x,b+x;a+b+x−1;dtα+dT+dt.Under the condition of formula A6, according to the mathematical expression formula ofLr,α,a,b|Θ=x,dx,dT (6), the middle of the two key variables A7 and A8, after reduction, we can get the ideal result, under the BP/NBD model; the mathematical expectation of overall θ hyperparameters under the posteriori probability has analytical solution, and the mathematic expression of the specific is as follows:(46)Eϒdt|Θ=x,dx,dT,r,α,a,b=a+b+x−1/a−11−α+dT/α+dT+dtrF12r+x,b+x;a+b+x−1;dt/α+dT+dt1+Ix>0a/b+x−1α+dT/α+dxr+x.


## 4. Experimental Analysis

### 4.1. Dataset of an Industry Competition

The premise of text mining of the entries requires complete project information:(1)
Most of the projects in the prototype design stage are in the initial stage, and there are some problems such as incomplete application materials. In this mining modeling process, the projects in the prototype design stage are eliminated.(2)
Projects without declaration materials (project documents) cannot pass the preliminary examination. In this mining modeling process, enterprises without declaration materials will be eliminated.Finally, 143 project contents are reserved for topic mining, and the case content and project output value are mined separately to extract important information and explore the application field, current situation, and key points of value promotion of industrial Internet.The main purpose of using this dataset is to adjust the discrete hyperparameters of the relevant prediction or classification model and compare the differences between the results before and after the adjustment, to form the experimental conclusion.


### 4.2. Participle

Jieba Chinese word segmentation component [10–12]is called, industrial Internet-related special words such as VR, AR, and smart Park are added into Jieba library, and precise mode is used to segment each document by default. In the process of word segmentation, function words and meaningless symbols are removed. To unify the expression of professional vocabulary, a thesaurus is set to merge synonyms to improve the effect of later topic extraction. For example, “artificial intelligence” and “Ai” are synonyms and merged into “Ai.” The unimportant words are filtered according to the part of speech, and the related words are finally reserved.


### 4.3. LDA Topic Extraction

The generated dictionary and corpus are in accordance with the input format of the model, and the open-source Gensim package [13] is used to construct the topic model and estimate the parameters. The default value of parameter selection is a = 0.37 and B = 0.02. The optimal topic number k is determined according to the perplexity of the model. In theory, the k-nearest neighbor with low confusion degree should be selected, but under the premise of small corpus database capacity, more topics may lead to overfitting phenomenon. Therefore, this time, the number of topics is adjusted by combining the coincidence degree of the visualization results. Finally, the optimal number of topics is 4, the optimal number of output value topics is 2, and there is no coincidence between topics.


### 4.4. Basic Model

According to the results of index analysis and LDA theme mining [14–16], the projects that are in the prototype design stage and lack of application materials are selected to be eliminated. Finally, 143 projects are retained to enter the modeling stage, of which 67.83% can enter the second round. Considering that there are many projects in the preliminary evaluation stage, which are not easily affected by external factors and can better reflect the development trend of current related industries, the paper uses XGBoost and multinomial NB to predict and model the preliminary evaluation results of 2020 related industry competition. The discrete hyperparameter indexes of the correlation model are the highest polynomial degree (multinomial NB) and the number of root nodes (XGBoost).


### 4.5. Evaluation Index

Because of the classification model prediction, the confusion matrix is selected as the basic evaluation index, and the evaluation index is derived from confusion matrix, as given in Table1.Table 1
Confusion matrix.
ClassificationForecast positive classForecast negative classActual positive classTP—true positiveFN—false negativeActual negative classFP—false positiveTN—true negative(1) Accuracy (ACC): accuracy = (TP + TN)/(TP + FN + FP + TN), the percentage of the correct results. (2) Precision: precision = TP/(TP + FP). In the records with positive prediction, how many are actually positive. (3) Recall rate (recall): recall = TP/(TP + FN); in the actual positive records, how many predictions are positive. (4) F1 score: 2/F1 = 1/precision + 1/recall.


### 4.6. Comparison of Experimental Results

In this modeling process, the polynomial naive Bayesian model (multinomial NB) and integrated model (XGBoost) are selected as training models (lack of specific mathematical description of the two methods). 70% of the samples are used for model training and 30% for testing. The parameters are adjusted in the training set dataset combined with threefold cross-validation, and “AUC” is used for training in the parameter adjustment process Figures1 and 2 show the effect of the model after adjusting the parameters, which has a certain reference value for the actual prediction performance of the model. On the test set, the effect of the model is as follows.Figure 1
Results using original discrete hyperparameters.Figure 2
Results after adjusting the discrete hyperparameters using the expected results of skew distribution.Through the comparison of experiments, we can find that(1)
From the classification prediction results of the whole model, the final result of XGBoost method is obviously better than the traditional Bayesian model in both the original configuration and the adjusted configuration(2)
The method of adjusting discrete hyperparameters based on skew distribution is effective. Through the empirical results of data experiments, the overall model, whether XGBoost or Bayesian model, achieves the improvement of relevant evaluation indicators under the new hyperparameter adjustment strategy.


## 4.1. Dataset of an Industry Competition

The premise of text mining of the entries requires complete project information:(1)
Most of the projects in the prototype design stage are in the initial stage, and there are some problems such as incomplete application materials. In this mining modeling process, the projects in the prototype design stage are eliminated.(2)
Projects without declaration materials (project documents) cannot pass the preliminary examination. In this mining modeling process, enterprises without declaration materials will be eliminated.Finally, 143 project contents are reserved for topic mining, and the case content and project output value are mined separately to extract important information and explore the application field, current situation, and key points of value promotion of industrial Internet.The main purpose of using this dataset is to adjust the discrete hyperparameters of the relevant prediction or classification model and compare the differences between the results before and after the adjustment, to form the experimental conclusion.


## 4.2. Participle

Jieba Chinese word segmentation component [10–12]is called, industrial Internet-related special words such as VR, AR, and smart Park are added into Jieba library, and precise mode is used to segment each document by default. In the process of word segmentation, function words and meaningless symbols are removed. To unify the expression of professional vocabulary, a thesaurus is set to merge synonyms to improve the effect of later topic extraction. For example, “artificial intelligence” and “Ai” are synonyms and merged into “Ai.” The unimportant words are filtered according to the part of speech, and the related words are finally reserved.


## 4.3. LDA Topic Extraction

The generated dictionary and corpus are in accordance with the input format of the model, and the open-source Gensim package [13] is used to construct the topic model and estimate the parameters. The default value of parameter selection is a = 0.37 and B = 0.02. The optimal topic number k is determined according to the perplexity of the model. In theory, the k-nearest neighbor with low confusion degree should be selected, but under the premise of small corpus database capacity, more topics may lead to overfitting phenomenon. Therefore, this time, the number of topics is adjusted by combining the coincidence degree of the visualization results. Finally, the optimal number of topics is 4, the optimal number of output value topics is 2, and there is no coincidence between topics.


## 4.4. Basic Model

According to the results of index analysis and LDA theme mining [14–16], the projects that are in the prototype design stage and lack of application materials are selected to be eliminated. Finally, 143 projects are retained to enter the modeling stage, of which 67.83% can enter the second round. Considering that there are many projects in the preliminary evaluation stage, which are not easily affected by external factors and can better reflect the development trend of current related industries, the paper uses XGBoost and multinomial NB to predict and model the preliminary evaluation results of 2020 related industry competition. The discrete hyperparameter indexes of the correlation model are the highest polynomial degree (multinomial NB) and the number of root nodes (XGBoost).


## 4.5. Evaluation Index

Because of the classification model prediction, the confusion matrix is selected as the basic evaluation index, and the evaluation index is derived from confusion matrix, as given in Table1.Table 1
Confusion matrix.
ClassificationForecast positive classForecast negative classActual positive classTP—true positiveFN—false negativeActual negative classFP—false positiveTN—true negative(1) Accuracy (ACC): accuracy = (TP + TN)/(TP + FN + FP + TN), the percentage of the correct results. (2) Precision: precision = TP/(TP + FP). In the records with positive prediction, how many are actually positive. (3) Recall rate (recall): recall = TP/(TP + FN); in the actual positive records, how many predictions are positive. (4) F1 score: 2/F1 = 1/precision + 1/recall.


## 4.6. Comparison of Experimental Results

In this modeling process, the polynomial naive Bayesian model (multinomial NB) and integrated model (XGBoost) are selected as training models (lack of specific mathematical description of the two methods). 70% of the samples are used for model training and 30% for testing. The parameters are adjusted in the training set dataset combined with threefold cross-validation, and “AUC” is used for training in the parameter adjustment process Figures1 and 2 show the effect of the model after adjusting the parameters, which has a certain reference value for the actual prediction performance of the model. On the test set, the effect of the model is as follows.Figure 1
Results using original discrete hyperparameters.Figure 2
Results after adjusting the discrete hyperparameters using the expected results of skew distribution.Through the comparison of experiments, we can find that(1)
From the classification prediction results of the whole model, the final result of XGBoost method is obviously better than the traditional Bayesian model in both the original configuration and the adjusted configuration(2)
The method of adjusting discrete hyperparameters based on skew distribution is effective. Through the empirical results of data experiments, the overall model, whether XGBoost or Bayesian model, achieves the improvement of relevant evaluation indicators under the new hyperparameter adjustment strategy.


## 5. Conclusion

Through the combination of mathematical analysis and empirical research, this study focuses on the optimization method of the hyperparameters of the relevant machine learning model under the condition of skew distribution, especially the discrete hyperparameters. Through strict mathematical assumptions and mathematical derivation, the expected value of the discrete hyperparameters is obtained, and then, the actual data test is carried out through the empirical dataset. The experimental results show that this heuristic parameter adjustment method is feasible and effective for XGBoost and Bayesian models.



---

*Source: 2835596-2022-08-09.xml*