# Zero-Inflated Patent Data Analysis Using Compound Poisson Models
## 3 Proposed Method
### Proposed Model  
Most elements of the document-word matrix have zero values because the number of words is much larger than the number of documents. That is, a word that appears even once in the whole document becomes one column in this matrix. In addition, the matrix has a frequency value only in the corresponding row and has a value of 0 in all other rows. Hence, the matrix data is a very sparse structure. This reduces the performance of predictive models. In previous studies, various models were proposed to solve the zero-inflated problem [9; 11; 12]. One of them is the zero-inflated count model such as zero-inflated Poisson (ZIP) and zero-inflated negative binomial (ZINB) models [9]. However, when the ratio of zeros included in the matrix becomes extremely high, the ZIP and ZINB models show the limitations to solve the sparsity problem. So, we propose a method to overcome the sparsity of text data using the compound Poisson linear model. In the proposed method, the matrix element is represented by (4):  
\[Y=0\text{ or }Y\in[1,\text{ }\infty) \tag{4}\]  
where \(Y\) is an element value in the matrix. If the word does not occur in document, the value of \(Y\) is 0. Theoretically, the range of possible frequency values (matrix elements) is from 1 to infinity. Since the variable Y takes so many numbers from 0 to infinity, we consider Y as a continuous variable in our study. In this paper, we use the keywords as the dependent and independent variables of statistical models. Additionally, we select a response variable from the keywords. The response variable is the target representing the domain of text documents. All variables other than the response can be predictors. We apply the compound distributions to analyze the sparse data with zero-inflated problem [14; 17; 36; 37; 38; 39; 40; 41]. In the compound distribution, a random variable \(Y\) is defined as (5) [17]:  
\[Y=\sum_{i=1}^{S}T_{i} \tag{5}\]  
where \(S\) is frequency value of occurred keyword distributed to the discrete probability distribution in (6) [14; 17; 42]:  
\[S\sim Poisson(m),\text{ }(m>0) \tag{6}\]  
That is, \(S\) follows Poisson distribution with parameter \(m\). In addition, \(T_{i}\) is the magnitude of \(i\)th keyword and distributed to gamma distribution as (7) [14; 17; 42]:  
\[T_{i}\sim Gamma(\alpha,\beta),\text{ }(\alpha>0,\text{ }\beta>0) \tag{7}\]  
In (7), \(\alpha\) and \(\beta\) are shape and scale parameters of gamma distribution. Additionally, the random variable \(T_{i}\) has the property of independent and identically distributed (iid). In (6) and (7), \(S\) and \(T_{i}\) are independent of each other. Using (5), (6) and (7), we perform a sparse text data analysis using compound Poisson generalized linear mixed model (GLMM). If \(Y\) follows the compound Poisson distribution belonging to the exponential dispersion model (EDM), then it is distributed to the probability density function in (8) [14]:  
\[\text{f}(Y=y|\mu,\sigma)=d_{p}(y,\phi)\exp(\frac{y\theta-c(\theta)}{\phi}), \text{ }\phi>0 \tag{8}\]where \(\phi\) and \(\theta\) are dispersion and canonical parameters, \(d_{p}(y,\phi)\) is a probability density function of EDM. Additionally, \(c(\theta)\) is a cumulant function similar to (9) [14]:  
\[c(\theta)= \begin{cases}\left(((1-p)\theta+1)^{\frac{2-p}{1-p}}-1\right)/(2-p)&, \ p\
eq 2\\ -\log(1-\theta)&,\ p=2\end{cases}  \tag{9}\]  
In (8) and (9), the compound Poisson distribution has \(p\) greater than \(1\) and less than \(2\). The GLMM is an extended version of GLM and includes an additional term and random effect as (10) [17; 39; 41; 43]:  
\[Y=X\beta+Zb+e,\ E(b)=E(e)=0 \tag{10}\]  
where \(X\) and \(Z\) are design matrices. Additionally, \(\beta\) and \(b\) are fixed and random effect parameters, respectively. The difference between a fixed effect and a random effect is whether there is an assumption about the distribution of parameters or not. When making an estimate, the fixed effect explicitly estimates the parameter, and in the case of the random effect, the distribution of the parameter is obtained.