nltk/nltk — Issue #125: Snowball stemmer for Russian muches apostrophes
Author: alexrudnick
Created: 2012-01-17 06:26:39+00:00
State: open

Version: 2.0b9
To reproduce:
>>> print stm.stem(u"Кот-д'Ивуару")
Output: кот-дьивуар
Notice the apostrophe being turned into Ь.
It shouldn't.

Migrated from http://code.google.com/p/nltk/issues/detail?id=593

earlier comments

PeMiStahl said, at 2010-09-12T16:30:37.000Z:

Hi,

I'm the one who ported the Snowball stemmers to NLTK. Therefore, I exactly followed the descriptions given by Martin Porter on his website. You can find the description of the Russian stemmer here: http://snowball.tartarus.org/algorithms/russian/stemmer.html

The apostrophe is turned into Ь because, before the actual stemming process starts, every Cyrillic letter is turned into its Roman transliterated form as stated by Mr Porter. After stemming, the transliterated form is turned back into the Cyrillic equivalent.

The problem now is the following: The stemming algorithm defined by Mr Porter assumes that an apostrophe may only appear in the Roman transliteration of word, but not in the original Cyrillic form. As I do not speak any Russian myself, I do not know whether an apostrophe can appear in a Russian word or not.

So, if you can tell me which positions an apostrophe can appear at in a Russian word, I might be able to adjust my code accordingly.

Thanks,
Peter

gmal.rom said, at 2010-09-18T00:29:00.000Z:

As it can appear only in loanwords, it's safely to assume it can appear within the root, not in first/last position and not in suffixes or flexions.

PeMiStahl said, at 2010-10-28T10:49:26.000Z:

Hi everyone,

I apologize for posting this so lately, but I moved to a new town and started my Master studies. Therefore, I just did not find the time to deal with this issue. Now I can provide a fix to the apostrophe problem. It was actually very easy to solve because I just had to add some new replace rules to the transliteration methods. The apostrophe is no longer turned into another character but stays the same during the stemming process.

Best,
Peter