Repository: PyThaiNLP/pythainlp (python)

Issue:
bug: Should raise error for using custom dict with unsupported word tokenization engine
### Description

Should raise an error when `custom_dict` is passed into `word_tokenizer` function but the selected engine is not supported custom_dict

### Expected results

Should either:
1. Raise `NotImplementedError` if the user passed in custom_dict to unsupported tokenizer engines
or
2. Raise a warning

### Current results

The current code completely ignores custom dict behavior when its engine not supported

https://github.com/PyThaiNLP/pythainlp/blob/3aa57c6bc62bdb91b6e1713303fcfa7e15855b1b/pythainlp/tokenize/core.py#L228-L235

### Steps to reproduce

```python
from pythainlp.tokenize import word_tokenize
from pythainlp.util import dict_trie


custom_dict = dict_trie(set(['tsts']))
word_tokenize('tstshel', custom_dict=custom_dict, engine = 'attacut')
```

### PyThaiNLP version

Commit hash 3aa57c6

### Python version

all

### Operating system and version

all

### More info

_No response_

### Possible solution

_No response_

### Files

_No response_

Pull request:
Fix custom dict error for unsupported tokenization engines
Fixes #1065

Add error handling for unsupported custom dictionaries in `word_tokenize` function.

* Add a check for unsupported engines in the `word_tokenize` function in `pythainlp/tokenize/core.py`.
* Raise a `NotImplementedError` if `custom_dict` is passed to an unsupported engine such as `attacut`, `icu`, `nercut`, `sefr_cut`, `tltk`, and `oskut`.
* Update the docstring for the `word_tokenize` function to reflect the changes.



Patch:
diff --git a/pythainlp/tokenize/core.py b/pythainlp/tokenize/core.py
index 1f8304c4..2fd58fed 100644
--- a/pythainlp/tokenize/core.py
+++ b/pythainlp/tokenize/core.py
@@ -111,10 +111,10 @@ def word_tokenize(
     :param str engine: name of the tokenizer to be used
     :param pythainlp.util.Trie custom_dict: dictionary trie (some engine may not support)
     :param bool keep_whitespace: True to keep whitespace, a common mark
-                                 for end of phrase in Thai.
-                                 Otherwise, whitespace is omitted.
+                                  for end of phrase in Thai.
+                                  Otherwise, whitespace is omitted.
     :param bool join_broken_num: True to rejoin formatted numeric that could be wrongly separated.
-                                 Otherwise, formatted numeric could be wrongly separated.
+                                  Otherwise, formatted numeric could be wrongly separated.
 
     :return: list of words
     :rtype: List[str]
@@ -221,6 +221,18 @@ def word_tokenize(
 
     segments = []
 
+    if custom_dict and engine in (
+        "attacut",
+        "icu",
+        "nercut",
+        "sefr_cut",
+        "tltk",
+        "oskut"
+    ):
+        raise NotImplementedError(
+            f"The {engine} engine does not support custom dictionaries."
+        )
+
     if engine in ("newmm", "onecut"):
         from pythainlp.tokenize.newmm import segment
 
@@ -366,7 +378,7 @@ def sent_tokenize(
             and ``wtp-large`` to use ``wtp-canine-s-12l`` model.
         * *whitespace+newline* - split by whitespace and newline.
         * *whitespace* - split by whitespace, specifically with \
-                         :class:`regex` pattern  ``r" +"``
+                          :class:`regex` pattern  ``r" +"``
     :Example:
 
     Split the text based on *whitespace*::
@@ -814,9 +826,9 @@ class Tokenizer:
                     used to create a trie, or an instantiated
                     :class:`pythainlp.util.Trie` object.
         :param str engine: choose between different options of tokenizer engines
-                           (i.e.  *newmm*, *mm*, *longest*, *deepcut*)
+                            (i.e.  *newmm*, *mm*, *longest*, *deepcut*)
         :param bool keep_whitespace: True to keep whitespace, a common mark
-                                    for end of phrase in Thai
+                                     for end of phrase in Thai
         """
         self.__trie_dict = Trie([])
         if custom_dict:


Test patch:
diff --git a/tests/core/test_tokenize.py b/tests/core/test_tokenize.py
index c1fd7ba0..b279931a 100644
--- a/tests/core/test_tokenize.py
+++ b/tests/core/test_tokenize.py
@@ -355,6 +355,11 @@ class TokenizeTestCase(unittest.TestCase):
             "ไฟ", word_tokenize("รถไฟฟ้า", custom_dict=dict_trie(["ไฟ"]))
         )
 
+        with self.assertRaises(NotImplementedError):
+            word_tokenize(
+                "รถไฟฟ้า", custom_dict=dict_trie(["ไฟ"]), engine="icu"
+            )
+
     def test_etcc(self):
         self.assertEqual(etcc.segment(None), [])
         self.assertEqual(etcc.segment(""), [])