fxsjy/jieba — pull request #160 by shenfeng (2014-05-20 01:52:12+00:00)
Title: Save word freqs in trie instead a dedicated map

freq可以保存在Trie里面，可以节约大约39M内存 (239m vs 200m)。

```sh
# 保存在 Trie里面
Partition of a set of 1400981 objects. Total size = 206311680 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 498458 36 165082352 80 165082352 80 dict (no owner)
1 512758 37 28714624 14 193796976 94 unicode
2 357899 26 8589576 4 202386552 98 float
3 14657 1 1272760 1 203659312 99 str
4 7412 1 602880 0 204262192 99 tuple
5 97 0 299032 0 204561224 99 dict of module
6 256 0 274048 0 204835272 99 dict of type
7 2054 0 262912 0 205098184 99 types.CodeType
8 1994 0 239280 0 205337464 100 function
9 256 0 227800 0 205565264 100 type
<117 more rows. Type e.g. '_.more' to view.>

# 保存在dict里面
Partition of a set of 1749883 objects. Total size = 239172392 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 498459 28 177665544 74 177665544 74 dict (no owner)
1 861769 49 49002952 20 226668496 95 unicode
2 357897 20 8589528 4 235258024 98 float
3 14615 1 1269856 1 236527880 99 str
4 7375 0 600144 0 237128024 99 tuple
5 97 0 299032 0 237427056 99 dict of module
6 255 0 273000 0 237700056 99 dict of type
7 2043 0 261504 0 237961560 99 types.CodeType
8 1985 0 238200 0 238199760 100 function
9 255 0 226896 0 238426656 100 type
<115 more rows. Type e.g. '_.more' to view.>

```

但由于Trie是python实现的，相比于C实现的dict，在lookup时，性能会低一些。 1.126s vs 1.555s

```python
import time
start = time.time()
for a in xrange(10000):
for w in jieba.cut(u'你认识那个和主席握手的的哥吗？他开一辆黑色的士。'):
pass

print time.time() - start

```