Just here to share a tool `fast-mosestokenizer`

mingruimingrui: <h1>Fast mosestokenizer</h1>
<h2>TLDR</h2>
<pre><code class="lang-auto">pip install fast-mosestokenizer
</code></pre>
<p>For a fast moses tokenizer.</p>
<h2>I’d like to share a tool</h2>
<p>Hey all,</p>
<p>Since the people here are all folks who dabble with NLP. I hope to share a tool that you guys would hopefully find useful.</p>
<p>Moses tokenization is a really popular tokenizer for many languages and gets used in research, business, and personal projects.</p>
<p>I wrapped the c++ tokenizer from the moses dedecoder package into a standalone library called <code>fast-mosestokenizer</code> <a href="https://github.com/mingruimingrui/fast-mosestokenizer" rel="nofollow noopener">https://github.com/mingruimingrui/fast-mosestokenizer</a>.<br>
The biggest benefit of using it is probably cross-language compatibility. In the package, an interface for python is provided and uploaded to PyPI.</p>
<p>Feel free to download and give it a try.</p>
<pre><code class="lang-auto">pip install fast-mosestokenizer
</code></pre>
<pre><code class="lang-auto">&gt;&gt;&gt; import mosestokenizer
&gt;&gt;&gt; tokenizer = mosestokenizer.MosesTokenizer(
    aggressive_dash_splits=True,
    refined_punct_splits=True,
)
&gt;&gt;&gt; tokenizer.tokenize("""
And you'd be right — in both cases.
But en can also be translated as "at,"
"about," "by," "on top of," "upon,"
"inside of" and other ways,
so it's use isn't as straightforward as it may appear.
""")
[
  'And', 'you', "'d", 'be', 'right', '—', 'in', 'both', 'cases', '.',
  'But', 'en', 'can', 'also', 'be', 'translated', 'as', '"', 'at', ',', '"',
  '"', 'about', ',', '"', '"', 'by', ',', '"', '"', 'on', 'top', 'of', ',', '"', '"', 'upon', ',', '"',
  '"', 'inside', 'of', '"', 'and', 'other', 'ways', ',',
  'so', 'it', "'s", 'use', 'is', "n't", 'as', 'straightforward', 'as', 'it', 'may', 'appear', '.'
]
</code></pre>
<p>The package is still in beta stages. You being an early adopter is going to help me out a lot in terms of further developing this package.</p>

ptrblck: <p>Thanks for sharing!<br>
CC <a class="mention" href="/u/thomas_wolf">@Thomas_Wolf</a> who might be interested in this. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

vdw: <p><a class="mention" href="/u/mingruimingrui">@mingruimingrui</a> thanks for sharing. May I ask how the tokenizer behaves in case in some example cases below:</p>
<ul>
<li>How can user mentions (e.g., <span class="mention">@DemoUser</span>)  or hashtags (e.g., <span class="hashtag">#helloworld</span>) tokenized</li>
<li>Does “This is…a test” get correctly tokenized?</li>
<li>Does "So much fun <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> Good night! get correctly tokenized (please assume no whitespaces before and after the emojis)</li>
<li>Does “Hi :-p” get correctly tokenized, i.e., the emoticon remains token “:-p”</li>
</ul>
<p>I work a lot with social media, and missing whitespaces, emojis as sentence separator, etc. are very common. Most off-the-shelf tokenizer struggle with most cases. And even some that, for example recognize, common emoticons such as “:-)” struggle with varitions like “:-)))”.</p>
<p>This is why I had to bend over backwards to write my own tokenizer that covers most common cases, but I’m happy to try others.</p>

mingruimingrui: <p>Let me try it out for you.</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import tokenizer

&gt;&gt;&gt; tokenizer = mosestokenizer.MosesTokenizer(
    lang='en',
    aggressive_dash_splits=True,
    refined_punct_splits=True,
)

&gt;&gt;&gt; cases = [
    '@DemoUser',
    '#helloworld',
    'This is…a test',
    'This is...a test'
    'So much fun🙂Goodnight!',
    'Hi :-p',
]

&gt;&gt;&gt; for case in cases:
&gt;&gt;&gt;     tokens = tokenizer.tokenize(case)
&gt;&gt;&gt;     print('{} -&gt; {} -&gt; {}'.format(
&gt;&gt;&gt;         case, tokens, tokenizer.detokenize(tokens)))
"@DemoUser -&gt; ['@', 'DemoUser'] -&gt; @ DemoUser"
"#helloworld -&gt; ['#', 'helloworld'] -&gt; # helloworld"
"This is…a test -&gt; ['This', 'is', '…', 'a', 'test'] -&gt; This is … a test"
"This is...a test -&gt; ['This', 'is', '...', 'a', 'test'] -&gt; This is... a test"
"So much fun🙂Goodnight! -&gt; ['So', 'much', 'fun', '🙂', 'Goodnight', '!'] -&gt; So much fun 🙂 Goodnight!"
"Hi :-p -&gt; ['Hi', ':', '@-@', 'p'] -&gt; Hi:-p"
</code></pre>
<p>Emoticon protection is probably out-of-scope of this package.<br>
There are also better ways to account for emoticons and special patterns.<br>
If you are looking for an out of the box solution without the need for any prior training<br>
or fine-tuning, <a href="https://github.com/cbaziotis/ekphrasis" rel="nofollow noopener">https://github.com/cbaziotis/ekphrasis</a> might be something worth looking at.</p>
<p>The main goal of this tokenizer is to perform similar tokenization to the original moses implementation (plus some slight adjustments).</p>
<p>Note that you can also provide protected regex patterns.</p>
<p>Using,</p>
<pre><code class="lang-auto">#protected_pattern.en
:-p
</code></pre>
<p>You get,</p>
<pre><code class="lang-auto">&gt;&gt;&gt; tokens = tokenizer.tokenize('Hi :-p')
&gt;&gt;&gt; print(tokens)
["Hi", ":-p"]
&gt;&gt;&gt; tokenizer.detokenize(tokens)
"Hi :-p"
</code></pre>

zeevso: <p><a class="mention" href="/u/mingruimingrui">@mingruimingrui</a> Thanks for sharing. I try to install using pip but with no success. I think that is because I use Windows. Any easy way for users who don’t want compiling and building from source code?</p>