Talk:Information gain (decision tree)

message: \{x\in Ex \wedge value(x,a)=v\}  and \{x\in Ex|value(x,a)=v\}  describes the same set, isn't it? Then it should be written identically also, otherwise it might confuse people. 84.57.82.107 08:49, 5 April 2007 (UTC)
thread_title:  Definition 

message: What is the definition of bullet? Is it the standard multiplication symbol like asterisk? (EasyWalker)  —Preceding unsigned comment added by 80.99.49.64 (talk) 21:11, 13 January 2009 (UTC)
thread_title:  Bullet 

message: Please change the notation for "examples" (Ex) to something different. Current notatoin is confusing because it looks as the expectation of x.  —Preceding unsigned comment added by 213.155.151.233 (talk) 14:48, 22 January 2009 (UTC)
thread_title:  Notation 

message: First, the notation is a little bad. The second equation for IG: 
 IG(T,a) = H(T) - \sum_{v \in vals(a)} \frac{| \lbrace \mathbf{x} \in T | x_a = v \rbrace |}{| T |} \ldots  
conflates the definition of \mathbf{x}, as \mathbf{x} is already defined above as the attributes which makes up T. T is a composed of attributes and an output value, y.  
Also, one only sums over all  v \in support(a)  when branching along all possible values for that attribute. It may turn out that  v \in \lbrace 1, 2, 3 \rbrace , but a branch occurs between a = 1 and  a \neq 1 . Thus I recommend making it clear that  v \in branches(a) . This produces the equation: 
The general and formal definitions appear to disagree: 
H(T) - H(T| x_a=v) = - \left(\sum_{t \in T} p(t) \log(p(t))\right) + \left( \sum_{t \in T} p(t|x_a=v) \log(p(t|x_a=v)) \right)
 
approximating p(t|x_a=v) \approx \frac{| \lbrace t \in T | x_a = v \rbrace |}{| T |} gives: 
H(T) - H(T| x_a=v) = - \left(\sum_{t \in T} p(t) \log(p(t))\right) + \left( \sum_{t \in T} \frac{| \lbrace t \in T | x_a = v \rbrace |}{| T |} \log\left( \frac{| \lbrace t \in T | x_a = v \rbrace |}{| T |} \right) \right)
Furthermore, summing over all branches produces: 
H(T) - \left( \sum_{v \in branches(a)} H(T|x_i = v) \right) = - \left(\sum_{t \in T} p(t) \log(p(t))\right) + \left( \sum_{v \in branches(a)} \left( \sum_{t \in T} \frac{| \lbrace t \in T | x_a = v \rbrace |}{| T |} \log\left( \frac{| \lbrace t \in T | x_a = v \rbrace |}{| T |} \right) \right) \right)
which still is not the equation given in the formal definition, but feels correct to me. 
While I am confused about what the correct form actually is, I see how to recover what is currently in the formal definition if we normalize (I am not even sure if it is correct to normalize) by the number of instances (aka samples) in each branch 
H(T) - \left( \sum_{v \in branches(a)} \frac{| \lbrace t \in T | x_a = v \rbrace |}{| T |} H(T|x_i = v) \right) = -\left(\sum_{t \in T} p(t) \log(p(t))\right) + \left( \sum_{v \in branches(a)} \frac{| \lbrace t \in T | x_a = v \rbrace |}{| T |} \left( \sum_{t \in T} \frac{| \lbrace t \in T | x_a = v \rbrace |}{| T |} \log\left( \frac{| \lbrace t \in T | x_a = v \rbrace |}{| T |} \right) \right) \right)
 
= - \left(\sum_{t \in T} p(t) \log(p(t))\right) + \left( \sum_{v \in branches(a)} \left( \sum_{t \in T} \left(\frac{| \lbrace t \in T | x_a = v \rbrace |}{| T |}\right)^2 \log\left( \frac{| \lbrace t \in T | x_a = v \rbrace |}{| T |} \right) \right) \right)
 
However, I repeat that normalizing here feels artificial. The information gain ratio normalizes another way (actually by dividing by something similar to the second term). 
So I wonder does:  
IG(T,a) = H(T) - \left( \sum_{v \in branches(a)} \frac{| \lbrace t \in T | x_a = v \rbrace |}{| T |} H(T|x_i = v) \right)  
or does  
IG(T,a) = H(T) - \left( \sum_{v \in branches(a)} H(T|x_i = v) \right)? 
The second one looks correct to me, so I suggest the formal equation should use 
IG(T,a) = H(T) - \left( \sum_{v \in branches(a)} H(T|x_i = v) \right)
 
IG(T,a) = - \left(\sum_{t \in T} p(t) \log(p(t))\right) + \left( \sum_{v \in branches(a)} \left( \sum_{t \in T} \frac{| \lbrace t \in T | x_a = v \rbrace |}{| T |} \log\left( \frac{| \lbrace t \in T | x_a = v \rbrace |}{| T |} \right) \right) \right)
 
Mouse7mouse9 23:47, 20 November 2014 (UTC)
replies: text: Mouse7mouse9 00:12, 21 November 2014 (UTC)
thread_title:  General and formal definitions disagree