Added ScalaCodeTokenizer.tokenize which takes a String instead of an IDocument.

ScalaCodeTokenizer.tokenize asked for a IDocument only to right away get the IDocument's contents and then never use it again. It makes more sense to just take the contents as a string directly, which is what this change does.
Also deprecated ScalaCodeTokenizer.tokenize(IDocument, Int, Int).

(cherry picked from commit 2754e16c032e3dd70bbd09a20c36ae1f23322274)
(cherry picked from commit 7ce9ecb59109ddd0ffed0fbf1d75af7bc503e21a)
Improved new ScalaCodeTokenizer.tokenize API to minimise allocation of string objects on caller side.

(cherry picked from commit 1c2db9513ffaea7c13b6260b87c6179c72e7895b)
(cherry picked from commit 89062ffc57e203055c3bfc38d03bcf8117faedd0)
Deprecated ScalaCodeScanner.tokenize(IDocument, Int, Int) incorrectly handled offset in generated tokens.

Also added a test.

Fixes bug introduced (into master) by SHA: e17b2ae97aaa5644d0cf1ad09ff6bc5ad039002a

(cherry picked from commit b9c41d8d2c5bddd88c471b297c53e2f3539014f2)
(cherry picked from commit bb403265ea05ae6187739cac23a32c6e5177fb88)
remove controversial & undocumented deprecation

(cherry picked from commit 9cd3d4f641aed2beab6d4133b8703c35893bd5ab)
(cherry picked from commit 0237fd65e733c6fdb31f05a1a40adb3e6c2f6342)
Merge pull request #523 from huitseeker/issue/backport-tokenize-1001866-juno

[Backport] Backport changes to the ScalaCodeScanner tokenizer