Repository: nltk/nltk

Issue:
HunposTagger error in Python3
Bug report from Tülin Erçelebi Ayyıldız:

> Dear hunpos authors,
> 
> I am currently trying to use hunpos tagger to tag a text file, however I get an error at the stage of hunpos.py run.
> 
> My configuration:
> OS : Windows 7-64bit
> python 3.4.1
> nltk 3.0.1
> 
> All "english.model", "hunpos-tag.exe" and "hunpos-train.exe" are located in "C:/Users" folder. My python code is as follows:
> ---

``` python
import nltk 
from nltk.tag.hunpos import HunposTagger
from nltk.tokenize import word_tokenize
corpus = "so how do i hunpos tag my ntuen ? i can't get the following code to work."

ht = HunposTagger('C:/Users/english.model','C:/Users/hunpos-tag.exe')
x=word_tokenize(corpus)
ht.tag(x)
```

> ---
> 
> When I run this module I get the following error:

```
Traceback (most recent call last):
  File "C:\Users\Tülin\Desktop\hunpos_deneme.py", line 12, in <module>
    ht.tag(x)
  File "C:\Python34\lib\site-packages\nltk\tag\hunpos.py", line 109, in tag
    self._hunpos.stdin.write(token + "\n")
TypeError: can't concat bytes to str
```

> I tried several things, but I could not successfully eliminate the problem and get a correct result.



Discussion:

Resolving patch:
diff --git a/nltk/tag/hunpos.py b/nltk/tag/hunpos.py
--- a/nltk/tag/hunpos.py
+++ b/nltk/tag/hunpos.py
@@ -3,8 +3,8 @@
 #
 # Copyright (C) 2001-2015 NLTK Project
 # Author: Peter Ljunglöf <peter.ljunglof@heatherleaf.se>
-#         David Nemeskey <nemeskeyd@gmail.com> (modifications)
-#         Attila Zseder <zseder@gmail.com> (modifications)
+#         Dávid Márk Nemeskey <nemeskeyd@gmail.com> (modifications)
+#         Attila Zséder <zseder@gmail.com> (modifications)
 # URL: <http://nltk.org/>
 # For license information, see LICENSE.TXT
 
@@ -71,14 +71,15 @@ def __init__(self, path_to_model, path_to_bin=None,
         hunpos_paths = list(map(os.path.expanduser, hunpos_paths))
 
         self._hunpos_bin = find_binary(
-                'hunpos-tag', path_to_bin,
-                env_vars=('HUNPOS_TAGGER',),
-                searchpath=hunpos_paths,
-                url=_hunpos_url,
-                verbose=verbose)
-
-        self._hunpos_model = find_file(path_to_model,
-                env_vars=('HUNPOS_TAGGER',), verbose=verbose)
+            'hunpos-tag', path_to_bin,
+            env_vars=('HUNPOS_TAGGER',),
+            searchpath=hunpos_paths,
+            url=_hunpos_url,
+            verbose=verbose
+        )
+
+        self._hunpos_model = find_file(
+            path_to_model, env_vars=('HUNPOS_TAGGER',), verbose=verbose)
         self._encoding = encoding
         self._hunpos = Popen([self._hunpos_bin, self._hunpos_model],
                              shell=False, stdin=PIPE, stdout=PIPE, stderr=PIPE)
@@ -106,14 +107,14 @@ def tag(self, tokens):
             assert "\n" not in token, "Tokens should not contain newlines"
             if isinstance(token, compat.text_type):
                 token = token.encode(self._encoding)
-            self._hunpos.stdin.write(token + "\n")
+            self._hunpos.stdin.write(token + b"\n")
         # We write a final empty line to tell hunpos that the sentence is finished:
-        self._hunpos.stdin.write("\n")
+        self._hunpos.stdin.write(b"\n")
         self._hunpos.stdin.flush()
 
         tagged_tokens = []
         for token in tokens:
-            tagged = self._hunpos.stdout.readline().strip().split("\t")
+            tagged = self._hunpos.stdout.readline().strip().split(b"\t")
             tag = (tagged[1] if len(tagged) > 1 else None)
             tagged_tokens.append((token, tag))
         # We have to read (and dismiss) the final empty line:


Test patch: