mishoo/UglifyJS — Issue #330: Tokenizer: Classification of null and Boolean values
Author: RGustBardon
Created: 2012-03-01 13:20:58+00:00
State: open

`./lib/parse-js.js`:

> ```javascript
function read_word() {
var word = read_name();
return !HOP(KEYWORDS, word)
? token("name", word)
: HOP(OPERATORS, word)
? token("operator", word)
: HOP(KEYWORDS_ATOM, word)
? token("atom", word)
: token("keyword", word);
};
```

Since `KEYWORDS` do not contain any elements of `KEYWORDS_ATOM`, I wonder if

```sh
$ echo "console.log(require('util').inspect(require('./lib/parse-js').parse(" \
"'null;false;true;undefined;eval;'), null, null));" | node
```

yielding

```json
[ 'toplevel',
[ [ 'stat', [ 'name', 'null' ] ],
[ 'stat', [ 'name', 'false' ] ],
[ 'stat', [ 'name', 'true' ] ],
[ 'stat', [ 'name', 'undefined' ] ],
[ 'stat', [ 'name', 'eval' ] ] ] ]
```

is intended?

The original tokenizer (a Common Lisp package) yields:

```lisp
(:TOPLEVEL
((:STAT (:ATOM :NULL)) (:STAT (:ATOM :FALSE)) (:STAT (:ATOM :TRUE))
(:STAT (:NAME "undefined")) (:STAT (:NAME "eval"))))
```

`null`, `false` and `true` are elements of `keywords` and `atom-keywords` in the source code written in Common Lisp.

Furthermore, I do not understand why `undefined` is *supposed* to be classified as an `atom`.