Questions and Answers on Likelihood

#Overview
There are already a number of useful resources explaining the concept of *likelihood*. In this post, I would like to share questions on *likelihood* I had initially, and answers to them for those who have questions as below:

>Q1. When we say *likelihood* should it always mean *likelihood function*?
Q2. Why the probability density function in Bayes' theorem is called *likelihood*?

This post is intended to be a complementary resource for those who are just learning the concept of *likelihood* and not fully comfortable with it. I would appreciate any comment or suggestion.


#Likelihood
Before going to questions, I first review *likelihood* here again. 

### Presumptions when we say *likelihood*

We assume a probability distribution of $x$ with parameter $\theta$ as $P_{\theta}(x)$, which is a *real* distribution we want to know. 

Whenever we say *likelihood*, we are already assuming followings:

- There is a series of $x$ observations as $X^{\prime} = (x_1, x_2, ... )$
- We are trying to explain the observation by a statistic model $p_{\theta}(x)$, which is our assumption to the real probability distribution.

###Likelihood (function)

*Likelihood function* is defined as below, when $p_{\theta}(x)$ is independent for any observation:

```math
L(\theta) = p_{\theta}(x_1) \cdot p_{\theta}(x_2) \cdot p_{\theta}(x_3) ... = 
```

Although the form of the equation is similar as probability density function, all $x$s in $L(\theta)$ are constants. Obviously, $L(\theta)$ is a function of $\theta$ and can be plotted on $L-\theta$ plane. $\theta$ can be a vector, but we assume it is a scalar for simplicity for now.

In the maximum likelihood estimate, we choose $\theta=\theta_{best}$ that maximize the likelihood. As it is usually said, the integral of the likelihood function is not necessary to be 1 since parameter $\theta$ is not a random variable of a probability distribution function.

#Question 1 - When we say *likelihood* should it always mean *likelihood function*?
When we search *likelihood*, it hits explanations on *likelihood function* and it is also mentioned that  *likelihood function* is abbreviated as *likelihood*. My question here was "So when we see *likelihood*, should we always imagine a function?" 

The answer is no. The value of likelihood function can also be called as *likelihood* as some references do. This value is called *likelihood* and of course it cannot be called as *likelihood function*. Thus, we cannot always assume *likelihood* as *likelihood function*. 

When $\theta$ is set to a certain value, say $\theta^{\prime}$, this results in one value $L(\theta^{\prime})$, which is a relative measure of how good the statistic model we assumed $p_{\theta}(x)$ is to explain the set of fixed observations $X^{\prime}$. 

To me, it is more clear to say:

>*likelihood* is a metric to relatively quantify how an assumed statistic model is good to explain given observations. A parametric function to calculate *likelihood* is called *likelihood function* or simply *likelihood* also. 

Likelifood function and its value are referred in the same word *likelihood*. This is similar as, for example, a kinetic energy of a point mass is written as a function of velocity but a certain value of the kinetic energy is also called kinetic energy.


#Question 2 - How the Likelihood definition relates to the likelihood term in Bayes' theorem?

In Bayes' theorem

```math
P(\theta|D) = \frac{P(D|\theta)}{P(D)}P(\theta)

```
The conditional probability $P(D|\theta)$ is called a *likelihood*. The prior $P(\theta)$ will be updated by multiplying the likelihood and normalized by *evidence* $P(D)$ to calculate the posterior $P(\theta|D)$. 

In text book examples of Baysian inference, $P(D|\theta)$ is sometime treated as a known conditional probability and does not seem to be a *likelihood* function. However, in general, this term is unknown. We usually have access to a series of observations $(d_1, d_2, ... )$ but do not know a mechanism, i.e. $P(D|\theta)$, that caused the observations.

Thus, $P(D|\theta)$ would be statistically modeled with parameters $\theta$ given an observation $D$. The form of $P(D|\theta)$ won't change since observations are assumed to be independent. This is consistent with the explanation of the likelihood function above.

### Maximum likelihood estimate (MLE) 
In Maximum Likelihood Estimate, the likelihood $P(D|\theta)$ as a function of $\theta$ is maximized with respect to $\theta$. One specific value of $\theta = \theta^{\prime}$ that maximize the likelihood is chosen to be a most possible value for the parameter. This is essentially equivalent to calculating the posterior $P(\theta|D)$ by assuming $\frac{P(\theta)}{P(D)}$ is constant and avoiding a prior belief for $\theta$ distribution to be a part of the estimate [4].

### Bayesian Estimate
In Bayesian Estimate, it treats $\theta$ as a random variable (= pips on a dice) defined by a probability distribution.  The value of $\theta$ is estimated by minimizing a loss function, such as variance [4, 5].

#References
[1] https://en.wikipedia.org/wiki/Likelihood_function
[2] http://www.genstat.net/statistics.html (Japanese)
[3] https://qiita.com/kenmatsu4/items/b28d1b3b3d291d0cc698 (Japanese)
[4] https://stats.stackexchange.com/questions/74082/what-is-the-difference-in-bayesian-estimate-and-maximum-likelihood-estimate
[5] https://medium.com/datadriveninvestor/maximum-likelihood-estimation-v-s-bayesian-estimation-bfac171a8b85