Added tokenizer.h and unit test
Merge pull request #41 from cesanta/tok_test

Added tokenizer.h and unit test
Moved private definitions to internal.h
Merge pull request #44 from cesanta/defs

Moved private definitions to internal.h
delete #include "tokenizer.h" from v7.c
Merge pull request #45 from vrz/master

delete #include "tokenizer.h" from v7.c
Remove undefined token

Ecma tests test for `var undefined = 1`.
The actual execution of this assignment is ignored,
but we should not treat this as reserved word token because it's not.

Not bothering fixing `parser.c` since reimplementation with AST is on its way.
Merge pull request #42 from cesanta/remove_undefined_token

Remove undefined token
WIP: AST based parser