ttezel/bayes — pull request #2 by kolarski (2014-07-16 15:46:26+00:00)
Title: Add Cyrilic Support

Sadly does not support UTF-8 (Cyrilic). The problem lies here:

```javascript
Naivebayes.prototype.tokenize = function (text) {
//remove punctuation from text - remove anything that isn't a word char or a space
var rgxPunctuation = /[^\w\s]/g

var sanitized = text.replace(rgxPunctuation, ' ')

return sanitized.split(/\s+/)
}
```

```javascript
var rgxPunctuation = /[^\w\s]/g
```
This matches only ASCII chars in JavaScript

You can test with

```javascript
tokenize("Надежда за обич еп.36 Тест")
```

returns:

```javasript
[ "", "36", "" ]
```

With the provided fix returns
```javascript
[ "Надежда", "за", "обич", "еп.36", "Тест" ]
```