Repository: WorksApplications/sudachi.rs (rust)

Issue:
Support unicode combining  symbols
e.g. in emojis like 👨‍👨‍👦‍👦 which are encoded as 7 codepoints: 4 people and 3 combining marks

It should be forbidden to begin OOVs on them and on the next symbol after them

## Variants to handle

Notation: A and C are some symbols, B is a combining mark.

1. ABC -> D (e.g. emoji)
2. AB -> C (most of unicode combining marks https://en.wikipedia.org/wiki/Combining_character)
3. ABC -> AC https://en.wikipedia.org/wiki/Zero-width_non-joiner

### Handling AB -> C

B should be marked as NOBOOV, B should be of ANY character category

### Handling ABC -> D

B and C should be marked as NOBOOV, B should be of ANY character category

### Handling ABC -> AC

B and C should be marked as NOBOOV, B should be of ANY character category (?)

Pull request:
support unicode combining symbols
fixes #136

As a side effect, `Config` API is becoming better for config object creation

Patch:
diff --git a/docs/errors_and_security.md b/docs/errors_and_security.md
new file mode 100644
index 0000000..a5346ed
--- /dev/null
+++ b/docs/errors_and_security.md
@@ -0,0 +1,33 @@
+# Errors
+
+## When to panic and when to error
+
+Sudachi.rs (as a library) must not panic or crash for any user input.
+Those situations should be returned as `Error`s.
+
+Sudachi.rs **can** panic or even produce undefined behavior for invalid binary dictionaries.
+There are rudimentary checks on the dictionary load, but the binary dictionaries are mostly trusted.
+Debug builds have checks on most operations that can produce an error or UB, 
+but some of them do not exist in release builds 
+(e.g. connection cost access can do out of bound accesses with invalid dictionaries).
+
+However, producing an invalid binary dictionary is considered a bug.
+Additionally, dictionary compilation should validate the csv data as much as possible (producing `Error`s).
+
+It is considered OK if Sudachi.rs panics for any input because of incorrect usage, but it should not be the case
+so Sudachi.rs panics only for some inputs.
+
+# Security
+
+Notes on security and error considerations for Sudachi.rs.
+
+## Threat model
+
+It should be safe to use Sudachi.rs with any user-inputted text.
+Crashes related to user input will be treated as bugs.
+
+On the other hand, maliciously crafted **binary** dictionaries are not treated as security threats.
+In the case of analyzing with possibly unsafe user-provided dictionaries, accept the dictionary in the csv form
+and compile it into the binary dictionary instead.
+Fortunately, binary dictionaries are mapped read-only and will not allow memory modifications, 
+but can allow data leaking in theory.
diff --git a/python/CHANGELOG.md b/python/CHANGELOG.md
index 688dafa..42a1e80 100644
--- a/python/CHANGELOG.md
+++ b/python/CHANGELOG.md
@@ -9,6 +9,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
   - `sudachipy.Tokenizer` will ignore the provided logger.
   - Ref: [#76]
 - `Morpheme.part_of_speech` method now returns Tuple of POS components instead of a list.
+- `Morpheme`/`MorphemeList` now have readable `__repr__` and `__str__`
+  - https://github.com/WorksApplications/sudachi.rs/pull/187
 
 ## [0.6.0] - 2021/10/11
 
diff --git a/python/py_src/sudachipy/resources/char.def b/python/py_src/sudachipy/resources/char.def
index b9728b7..90985b4 100644
--- a/python/py_src/sudachipy/resources/char.def
+++ b/python/py_src/sudachipy/resources/char.def
@@ -133,7 +133,6 @@ CYRILLIC        1 1 0
 0x2000..0x206F  SYMBOL # General Punctuation
 0x2070..0x209F  NUMERIC # Superscripts and Subscripts
 0x20A0..0x20CF  SYMBOL # Currency Symbols
-0x20D0..0x20FF  SYMBOL # Combining Diaritical Marks for Symbols
 0x2100..0x214F  SYMBOL # Letterlike Symbols
 0x2150..0x218F  NUMERIC # Number forms
 0x2100..0x214B  SYMBOL # Letterlike Symbols
@@ -158,10 +157,23 @@ CYRILLIC        1 1 0
 0xFE30..0xFE4F  SYMBOL # CJK Compatibility Forms
 0xFE50..0xFE6B  SYMBOL # Small Form Variants
 
-# added 2006/3/13 
+# added 2006/3/13
 0x3007 SYMBOL KANJINUMERIC
 
 # added 2018/11/30
 0x309b..0x309c HIRAGANA KATAKANA # voiced/semi-voiced sound marks
 
+# Unicode combining symbols 2021/12/03
+# from https://en.wikipedia.org/wiki/Combining_character
+0x0300..0x036F ALL NOOOVBOW # Combining Diacritical Marks
+0x1AB0..0x1AFF ALL NOOOVBOW # Combining Diacritical Marks Extended
+0x1DC0..0x1DFF ALL NOOOVBOW # Combining Diacritical Marks Supplement
+0x20D0..0x20FF ALL NOOOVBOW # Combining Diacritical Marks for Symbols
+0xFE20..0xFE2F ALL NOOOVBOW # Combining Half Marks
+0xFE00..0xFE0F ALL NOOOVBOW # https://codepoints.net/variation_selectors
+0x1F3FB..0x1F3FE ALL NOOOVBOW # emoji skin tone modifiers https://codepoints.net/U+1F3FF
+
+# Combination marks
+0x200C..0x200D ALL NOOOVBOW2 # Zero Width Non-Joiner/Joiner
+
 # END OF TABLE
diff --git a/python/src/morpheme.rs b/python/src/morpheme.rs
index 237b6bf..99d273d 100644
--- a/python/src/morpheme.rs
+++ b/python/src/morpheme.rs
@@ -14,10 +14,11 @@
  *  limitations under the License.
  */
 
+use std::fmt::Write;
 use std::ops::Deref;
 use std::sync::Arc;
 
-use pyo3::exceptions::{self, PyException};
+use pyo3::exceptions::{self, PyException, PyIndexError};
 use pyo3::prelude::*;
 use pyo3::types::{PyList, PyString, PyTuple, PyType};
 
@@ -107,8 +108,8 @@ impl PyMorphemeListWrapper {
         }
 
         if idx < 0 || len <= idx {
-            return Err(PyErr::new::<exceptions::PyIndexError, _>(format!(
-                "morphemelist index out of range: the len is {} but the index is {}",
+            return Err(PyIndexError::new_err(format!(
+                "MorphemeList index out of range: the len is {} but the index is {}",
                 list.size(py),
                 idx
             )));
@@ -123,7 +124,38 @@ impl PyMorphemeListWrapper {
     }
 
     fn __str__<'py>(&'py self, py: Python<'py>) -> &PyString {
-        PyString::new(py, self.internal(py).surface().deref())
+        // do a simple tokenization __str__
+        let list = self.internal(py);
+        let mut result = String::with_capacity(list.surface().len() * 2);
+        let nmorphs = list.len();
+        for (i, m) in list.iter().enumerate() {
+            result.push_str(m.surface().deref());
+            if i + 1 != nmorphs {
+                result.push_str(" ");
+            }
+        }
+        PyString::new(py, result.as_str())
+    }
+
+    fn __repr__(slf: Py<PyMorphemeListWrapper>, py: Python) -> PyResult<&PyString> {
+        let self_ref = slf.borrow(py);
+        let list = self_ref.internal(py);
+        let mut result = String::with_capacity(list.surface().len() * 10);
+        result.push_str("<MorphemeList[\n");
+        let nmorphs = list.len();
+        for i in 0..nmorphs {
+            result.push_str("  ");
+            let pymorph = PyMorpheme {
+                list: slf.clone_ref(py),
+                index: i,
+            };
+            pymorph
+                .write_repr(py, &mut result)
+                .map_err(|_| PyException::new_err("format failed"))?;
+            result.push_str(",\n");
+        }
+        result.push_str("]>");
+        Ok(PyString::new(py, result.as_str()))
     }
 
     fn __iter__(slf: Py<Self>) -> PyMorphemeIter {
@@ -207,14 +239,24 @@ impl PyMorpheme {
         let morph = unsafe { std::mem::transmute(list.internal(py).get(self.index)) };
         MorphemeRef { list, morph }
     }
+
+    fn write_repr<'py, W: Write>(&'py self, py: Python<'py>, out: &mut W) -> std::fmt::Result {
+        // per https://github.com/WorksApplications/SudachiPy/pull/166#issuecomment-932043063
+        let mrp = self.morph(py);
+        let surf = mrp.surface();
+        write!(
+            out,
+            "<Morpheme({}, {}:{}, {})>",
+            surf.deref(),
+            mrp.begin_c(),
+            mrp.end_c(),
+            mrp.word_id()
+        )
+    }
 }
 
 #[pymethods]
 impl PyMorpheme {
-    fn __str__<'py>(&'py self, py: Python<'py>) -> PyObject {
-        self.surface(py)
-    }
-
     /// Returns the begin index of this in the input text
     #[pyo3(text_signature = "($self)")]
     fn begin(&self, py: Python) -> usize {
@@ -358,4 +400,15 @@ impl PyMorpheme {
         let m = self.morph(py);
         m.end_c() - m.begin_c()
     }
+
+    pub fn __str__<'py>(&'py self, py: Python<'py>) -> PyObject {
+        self.surface(py)
+    }
+
+    pub fn __repr__<'py>(&'py self, py: Python<'py>) -> PyResult<String> {
+        let mut result = String::new();
+        self.write_repr(py, &mut result)
+            .map_err(|_| PyException::new_err("failed to format repr"))?;
+        Ok(result)
+    }
 }
diff --git a/python/src/pos_matcher.rs b/python/src/pos_matcher.rs
index 2618295..8ac83d0 100644
--- a/python/src/pos_matcher.rs
+++ b/python/src/pos_matcher.rs
@@ -121,7 +121,7 @@ impl PyPosMatcher {
     }
 
     pub fn __str__(&self) -> String {
-        format!("PosMatcher<{} pos>", self.matcher.num_entries())
+        format!("<PosMatcher:{} pos>", self.matcher.num_entries())
     }
 
     pub fn __iter__(&self) -> PyPosIter {
diff --git a/resources/char.def b/resources/char.def
index b9728b7..83e789a 100644
--- a/resources/char.def
+++ b/resources/char.def
@@ -133,7 +133,6 @@ CYRILLIC        1 1 0
 0x2000..0x206F  SYMBOL # General Punctuation
 0x2070..0x209F  NUMERIC # Superscripts and Subscripts
 0x20A0..0x20CF  SYMBOL # Currency Symbols
-0x20D0..0x20FF  SYMBOL # Combining Diaritical Marks for Symbols
 0x2100..0x214F  SYMBOL # Letterlike Symbols
 0x2150..0x218F  NUMERIC # Number forms
 0x2100..0x214B  SYMBOL # Letterlike Symbols
@@ -164,4 +163,17 @@ CYRILLIC        1 1 0
 # added 2018/11/30
 0x309b..0x309c HIRAGANA KATAKANA # voiced/semi-voiced sound marks
 
+# Unicode combining symbols 2021/12/03
+# from https://en.wikipedia.org/wiki/Combining_character
+0x0300..0x036F ALL NOOOVBOW # Combining Diacritical Marks
+0x1AB0..0x1AFF ALL NOOOVBOW # Combining Diacritical Marks Extended
+0x1DC0..0x1DFF ALL NOOOVBOW # Combining Diacritical Marks Supplement
+0x20D0..0x20FF ALL NOOOVBOW # Combining Diacritical Marks for Symbols
+0xFE20..0xFE2F ALL NOOOVBOW # Combining Half Marks
+0xFE00..0xFE0F ALL NOOOVBOW # https://codepoints.net/variation_selectors
+0x1F3FB..0x1F3FE ALL NOOOVBOW # emoji skin tone modifiers https://codepoints.net/U+1F3FF
+
+# Combination marks
+0x200C..0x200D ALL NOOOVBOW2 # Zero Width Non-Joiner/Joiner
+
 # END OF TABLE
diff --git a/sudachi/src/analysis/stateful_tokenizer.rs b/sudachi/src/analysis/stateful_tokenizer.rs
index b00b504..80ffa98 100644
--- a/sudachi/src/analysis/stateful_tokenizer.rs
+++ b/sudachi/src/analysis/stateful_tokenizer.rs
@@ -220,16 +220,13 @@ impl<D: DictionaryAccess> StatefulTokenizer<D> {
         lattice.reset(input.current_chars().len());
 
         for (ch_off, &byte_off) in input.curr_byte_offsets().iter().enumerate() {
-            if !input.can_bow(byte_off) {
-                continue;
-            }
-
             if !lattice.has_previous_node(ch_off) {
                 continue;
             }
 
             let mut has_word = false;
             for e in dict.lexicon().lookup(input_bytes, byte_off) {
+                // do we really need input.can_bow condition?
                 if (e.end < input_bytes.len()) && !input.can_bow(e.end) {
                     continue;
                 }
@@ -248,7 +245,10 @@ impl<D: DictionaryAccess> StatefulTokenizer<D> {
             }
 
             // OOV
-            if !input.cat_at_char(ch_off).contains(CategoryType::NOOOVBOW) {
+            if !input
+                .cat_at_char(ch_off)
+                .intersects(CategoryType::NOOOVBOW | CategoryType::NOOOVBOW2)
+            {
                 for oov_provider in dict.oov_provider_plugins() {
                     oov_provider.get_oov(&input, ch_off, has_word, oovs)?;
                 }
@@ -261,7 +261,7 @@ impl<D: DictionaryAccess> StatefulTokenizer<D> {
             if !has_word {
                 dict.oov_provider_plugins()
                     .last()
-                    .unwrap()
+                    .unwrap() // JapaneseDictionary
                     .get_oov(&input, ch_off, has_word, oovs)?;
                 // use last oov_provider as default
                 for node in oovs.drain(..) {
@@ -271,7 +271,7 @@ impl<D: DictionaryAccess> StatefulTokenizer<D> {
             }
 
             if !has_word {
-                panic!("no morpheme found at {}", byte_off);
+                return Err(SudachiError::EosBosDisconnect);
             }
         }
         lattice.connect_eos(dict.grammar().conn_matrix())?;
diff --git a/sudachi/src/config.rs b/sudachi/src/config.rs
index 6ec273a..f3bf78d 100644
--- a/sudachi/src/config.rs
+++ b/sudachi/src/config.rs
@@ -17,7 +17,7 @@
 use std::env::current_exe;
 use std::fs::File;
 use std::io::BufReader;
-use std::path::PathBuf;
+use std::path::{Path, PathBuf};
 
 use lazy_static::lazy_static;
 use serde::Deserialize;
@@ -48,7 +48,7 @@ pub enum ConfigError {
 }
 
 /// Setting data loaded from config file
-#[derive(Debug, Default)]
+#[derive(Debug, Default, Clone)]
 pub struct Config {
     pub resource_dir: PathBuf,
     pub system_dict: Option<PathBuf>,
@@ -66,7 +66,7 @@ pub struct Config {
 /// For plugins, refer to each plugin.
 #[allow(non_snake_case)]
 #[derive(Deserialize, Debug)]
-pub struct RawConfig {
+pub struct ConfigBuilder {
     resourcePath: Option<PathBuf>,
     systemDict: Option<PathBuf>,
     userDict: Option<Vec<PathBuf>>,
@@ -78,60 +78,111 @@ pub struct RawConfig {
     pathRewritePlugin: Option<Vec<Value>>,
 }
 
-impl Config {
-    pub fn new(
-        config_file: Option<PathBuf>,
-        resource_dir: Option<PathBuf>,
-        dictionary_path: Option<PathBuf>,
-    ) -> Result<Self, ConfigError> {
+impl ConfigBuilder {
+    pub fn from_file(config_file: &Path) -> Result<Self, ConfigError> {
+        let file = File::open(config_file)?;
+        let reader = BufReader::new(file);
+        serde_json::from_reader(reader).map_err(|e| e.into())
+    }
+
+    pub fn from_bytes(data: &[u8]) -> Result<Self, ConfigError> {
+        serde_json::from_slice(data).map_err(|e| e.into())
+    }
+
+    pub fn empty() -> Self {
+        serde_json::from_slice(b"{}").unwrap()
+    }
+
+    pub fn system_dict(mut self, dict: impl Into<PathBuf>) -> Self {
+        self.systemDict = Some(dict.into());
+        self
+    }
+
+    pub fn user_dict(mut self, dict: impl Into<PathBuf>) -> Self {
+        let dicts = match self.userDict.as_mut() {
+            None => {
+                self.userDict = Some(Default::default());
+                self.userDict.as_mut().unwrap()
+            }
+            Some(dicts) => dicts,
+        };
+        dicts.push(dict.into());
+        self
+    }
+
+    pub fn resource_path(mut self, path: impl Into<PathBuf>) -> Self {
+        self.resourcePath = Some(path.into());
+        self
+    }
+
+    pub fn build(self) -> Config {
         let src_root_path = PathBuf::from(env!("CARGO_MANIFEST_DIR"));
         let default_resource_dir_path = src_root_path.join("..").join(DEFAULT_RESOURCE_DIR);
 
-        // prioritize arg (cli option) > default
-        let config_file = match config_file {
-            Some(v) => v,
-            None => default_resource_dir_path.join(DEFAULT_SETTING_FILE),
-        };
-        let file = File::open(config_file)?;
-        let reader = BufReader::new(file);
-        let raw_config: RawConfig = serde_json::from_reader(reader)?;
+        let resource_dir = self.resourcePath.unwrap_or(default_resource_dir_path);
 
-        // prioritize arg (cli option) > config file > default
-        let resource_dir = resource_dir
-            .or_else(|| raw_config.resourcePath.clone())
-            .unwrap_or_else(|| default_resource_dir_path);
+        let system_dict = self
+            .systemDict
+            .clone()
+            .map(|p| Config::join_if_relative(&resource_dir, p));
 
-        // prioritize arg (cli option) > config file
-        let system_dict = dictionary_path.or_else(|| {
-            raw_config
-                .systemDict
-                .clone()
-                .map(|p| Config::join_if_relative(&resource_dir, p))
-        });
-        let user_dicts = raw_config
+        let user_dicts = self
             .userDict
             .unwrap_or(Vec::new())
             .into_iter()
             .map(|p| Config::join_if_relative(&resource_dir, p))
             .collect();
+
         let character_definition_file = Config::join_if_relative(
             &resource_dir,
-            raw_config
-                .characterDefinitionFile
+            self.characterDefinitionFile
                 .unwrap_or(PathBuf::from(DEFAULT_CHAR_DEF_FILE)),
         );
 
-        Ok(Config {
+        Config {
             resource_dir,
             system_dict,
             user_dicts,
             character_definition_file,
 
-            connection_cost_plugins: raw_config.connectionCostPlugin.unwrap_or(Vec::new()),
-            input_text_plugins: raw_config.inputTextPlugin.unwrap_or(Vec::new()),
-            oov_provider_plugins: raw_config.oovProviderPlugin.unwrap_or(Vec::new()),
-            path_rewrite_plugins: raw_config.pathRewritePlugin.unwrap_or(Vec::new()),
-        })
+            connection_cost_plugins: self.connectionCostPlugin.unwrap_or(Vec::new()),
+            input_text_plugins: self.inputTextPlugin.unwrap_or(Vec::new()),
+            oov_provider_plugins: self.oovProviderPlugin.unwrap_or(Vec::new()),
+            path_rewrite_plugins: self.pathRewritePlugin.unwrap_or(Vec::new()),
+        }
+    }
+}
+
+impl Config {
+    pub fn new(
+        config_file: Option<PathBuf>,
+        resource_dir: Option<PathBuf>,
+        dictionary_path: Option<PathBuf>,
+    ) -> Result<Self, ConfigError> {
+        let src_root_path = PathBuf::from(env!("CARGO_MANIFEST_DIR"));
+        let default_resource_dir_path = src_root_path.join("..").join(DEFAULT_RESOURCE_DIR);
+
+        // prioritize arg (cli option) > default
+        let config_file = match config_file {
+            Some(v) => v,
+            None => default_resource_dir_path.join(DEFAULT_SETTING_FILE),
+        };
+
+        let raw_config = ConfigBuilder::from_file(&config_file)?;
+
+        // prioritize arg (cli option) > config file
+        let raw_config = match resource_dir {
+            None => raw_config,
+            Some(p) => raw_config.resource_path(p),
+        };
+
+        // prioritize arg (cli option) > config file
+        let raw_config = match dictionary_path {
+            None => raw_config,
+            Some(p) => raw_config.system_dict(p),
+        };
+
+        Ok(raw_config.build())
     }
 
     /// Creates a minimal config with the provided resource directory
diff --git a/sudachi/src/dic/category_type.rs b/sudachi/src/dic/category_type.rs
index 4407cda..189df5f 100644
--- a/sudachi/src/dic/category_type.rs
+++ b/sudachi/src/dic/category_type.rs
@@ -54,11 +54,13 @@ bitflags! {
         const USER3 = (1 << 13);
         /** User defined category. */
         const USER4 = (1 << 14);
-        /** Characters that cannot be the beginning of word */
-        const NOOOVBOW = (1 << 31);
+        /** This character cannot be the beginning of an OOV word */
+        const NOOOVBOW = (1 << 30);
+        /** This and next characters cannot be the beginning of an OOV word */
+        const NOOOVBOW2 = (1 << 31);
 
-        /** All categories at once except NOOOVBOW */
-        const ALL = !(1 << 31);
+        /** All categories at once except NOOOVBOW/2 */
+        const ALL = 0b00111111_11111111_11111111_11111111;
     }
 }
 
@@ -122,6 +124,7 @@ impl FromStr for CategoryType {
             "USER3" => Ok(Self::USER3),
             "USER4" => Ok(Self::USER4),
             "NOOOVBOW" => Ok(Self::NOOOVBOW),
+            "NOOOVBOW2" => Ok(Self::NOOOVBOW2),
             "ALL" => Ok(Self::ALL),
             _ => Err(SudachiError::InvalidCharacterCategoryType(String::from(s))),
         }
diff --git a/sudachi/src/dic/connect.rs b/sudachi/src/dic/connect.rs
index a9a18f4..957ecd6 100644
--- a/sudachi/src/dic/connect.rs
+++ b/sudachi/src/dic/connect.rs
@@ -34,7 +34,7 @@ impl<'a> ConnectionMatrix<'a> {
 
         let end = offset + size;
         if end > data.len() {
-            return Err(SudachiError::InvalidDictionaryGrammar);
+            return Err(SudachiError::InvalidDictionaryGrammar.with_context("connection matrix"));
         }
 
         Ok(ConnectionMatrix {
diff --git a/sudachi/src/dic/dictionary.rs b/sudachi/src/dic/dictionary.rs
index 2d5ea7e..87a9b8b 100644
--- a/sudachi/src/dic/dictionary.rs
+++ b/sudachi/src/dic/dictionary.rs
@@ -59,7 +59,7 @@ fn map_file(path: &Path) -> SudachiResult<Storage> {
 
 fn load_system_dic(cfg: &Config) -> SudachiResult<Storage> {
     match &cfg.system_dict {
-        Some(p) => map_file(p),
+        Some(p) => map_file(p).map_err(|e| e.with_context(p.as_os_str().to_string_lossy())),
         None => return Err(ConfigError(MissingArgument(String::from("system_dict")))),
     }
 }
@@ -70,7 +70,10 @@ impl JapaneseDictionary {
         let mut sb = SudachiDicData::new(load_system_dic(cfg)?);
 
         for udic in cfg.user_dicts.iter() {
-            sb.add_user(map_file(udic.as_path())?)
+            sb.add_user(
+                map_file(udic.as_path())
+                    .map_err(|e| e.with_context(udic.as_os_str().to_string_lossy()))?,
+            )
         }
 
         Self::from_cfg_storage(cfg, sb)
diff --git a/sudachi/src/dic/grammar.rs b/sudachi/src/dic/grammar.rs
index f958a76..3e9f3c3 100644
--- a/sudachi/src/dic/grammar.rs
+++ b/sudachi/src/dic/grammar.rs
@@ -53,8 +53,8 @@ impl<'a> Grammar<'a> {
     /// buf: reference to the dictionary bytes
     /// offset: offset to the grammar section in the buf
     pub fn parse(buf: &[u8], offset: usize) -> SudachiResult<Grammar> {
-        let (rest, (pos_list, left_id_size, right_id_size)) =
-            grammar_parser(buf, offset).map_err(|_| SudachiError::InvalidDictionaryGrammar)?;
+        let (rest, (pos_list, left_id_size, right_id_size)) = grammar_parser(buf, offset)
+            .map_err(|e| SudachiError::InvalidDictionaryGrammar.with_context(e.to_string()))?;
 
         let connect_table_offset = buf.len() - rest.len();
         let storage_size =
diff --git a/sudachi/src/error.rs b/sudachi/src/error.rs
index c22c02f..4e43b64 100644
--- a/sudachi/src/error.rs
+++ b/sudachi/src/error.rs
@@ -15,6 +15,7 @@
  */
 
 use std::fmt::Debug;
+use std::io::Error;
 use thiserror::Error;
 
 use crate::config::ConfigError;
@@ -30,8 +31,17 @@ pub type SudachiResult<T> = Result<T, SudachiError>;
 #[derive(Error, Debug)]
 #[non_exhaustive]
 pub enum SudachiError {
-    #[error("IO Error: {0}")]
-    Io(#[from] std::io::Error),
+    #[error("{context}: {cause}")]
+    ErrWithContext {
+        context: String,
+        cause: Box<SudachiError>,
+    },
+
+    #[error("{context}: {cause}")]
+    Io {
+        cause: std::io::Error,
+        context: String,
+    },
 
     #[error("Parse Int Error")]
     ParseIntError(#[from] std::num::ParseIntError),
@@ -42,9 +52,6 @@ pub enum SudachiError {
     #[error("Regex error")]
     RegexError(#[from] fancy_regex::Error),
 
-    #[error("Libloading Error: {0}")]
-    Libloading(#[from] libloading::Error),
-
     #[error("Error from nom {0}")]
     NomParseError(String),
 
@@ -87,21 +94,6 @@ pub enum SudachiError {
     #[error("Invalid range: {0}..{1}")]
     InvalidRange(usize, usize),
 
-    #[error("Missing dictionary trie")]
-    MissingDictionaryTrie,
-
-    #[error("Missing latice path")]
-    MissingLaticePath,
-
-    #[error("Missing part of speech")]
-    MissingPartOfSpeech,
-
-    #[error("Missing word_id")]
-    MissingWordId,
-
-    #[error("Missing word_info")]
-    MissingWordInfo,
-
     #[error("No out of vocabulary plugin provided")]
     NoOOVPluginProvided,
 
@@ -115,6 +107,30 @@ pub enum SudachiError {
     MorphemeListBorrowed,
 }
 
+impl From<std::io::Error> for SudachiError {
+    fn from(e: Error) -> Self {
+        SudachiError::Io {
+            cause: e,
+            context: String::from("IO Error"),
+        }
+    }
+}
+
+impl SudachiError {
+    pub fn with_context<S: Into<String>>(self, ctx: S) -> Self {
+        match self {
+            SudachiError::Io { cause, .. } => SudachiError::Io {
+                cause,
+                context: ctx.into(),
+            },
+            cause => SudachiError::ErrWithContext {
+                cause: Box::new(cause),
+                context: ctx.into(),
+            },
+        }
+    }
+}
+
 pub type SudachiNomResult<I, O> = nom::IResult<I, O, SudachiNomError<I>>;
 
 /// Custum nom error
diff --git a/sudachi/src/input_text/buffer/mod.rs b/sudachi/src/input_text/buffer/mod.rs
index 91dfa28..db2996a 100644
--- a/sudachi/src/input_text/buffer/mod.rs
+++ b/sudachi/src/input_text/buffer/mod.rs
@@ -144,6 +144,7 @@ impl InputBuffer {
         let non_starting = CategoryType::ALPHA | CategoryType::GREEK | CategoryType::CYRILLIC;
         let mut prev_cat = CategoryType::empty();
         self.mod_bow.resize(self.modified.len(), false);
+        let mut next_bow = true;
 
         for (chidx, (bidx, ch)) in self.modified.char_indices().enumerate() {
             self.mod_chars.push(ch);
@@ -155,12 +156,26 @@ impl InputBuffer {
             last_offset = bidx;
             last_chidx = chidx;
 
-            // BOW logic: for special cases check if the previous char is compatible
-            self.mod_bow[bidx] = if cat.intersects(non_starting) {
+            let can_bow = if !next_bow {
+                // this char was forbidden by the previous one
+                next_bow = true;
+                false
+            } else if cat.intersects(CategoryType::NOOOVBOW2) {
+                // this rule is stronger than the next one and must come before
+                // this and next are forbidden
+                next_bow = false;
+                false
+            } else if cat.intersects(CategoryType::NOOOVBOW) {
+                // this char is forbidden
+                false
+            } else if cat.intersects(non_starting) {
+                // the previous char is compatible
                 !cat.intersects(prev_cat)
             } else {
                 true
             };
+
+            self.mod_bow[bidx] = can_bow;
             prev_cat = cat;
         }
         // trailing indices for the last codepoint
diff --git a/sudachi/src/plugin/loader.rs b/sudachi/src/plugin/loader.rs
index 3dc2058..726f5f5 100644
--- a/sudachi/src/plugin/loader.rs
+++ b/sudachi/src/plugin/loader.rs
@@ -127,7 +127,8 @@ impl<'a, T: PluginCategory + ?Sized> PluginLoader<'a, T> {
                 self.load_plugin_from_dso(&candidates)?
             };
 
-        <T as PluginCategory>::do_setup(&mut plugin, plugin_cfg, &self.cfg, &self.grammar)?;
+        <T as PluginCategory>::do_setup(&mut plugin, plugin_cfg, &self.cfg, &self.grammar)
+            .map_err(|e| e.with_context(format!("plugin {} setup", name)))?;
         self.plugins.push(plugin);
         Ok(())
     }
@@ -143,7 +144,7 @@ impl<'a, T: PluginCategory + ?Sized> PluginLoader<'a, T> {
         resolved
     }
 
-    fn try_load_library_from(candidates: &[String]) -> SudachiResult<Library> {
+    fn try_load_library_from(candidates: &[String]) -> SudachiResult<(Library, &str)> {
         if candidates.is_empty() {
             return Err(SudachiError::PluginError(PluginError::InvalidDataFormat(
                 "No candidates to load library".to_owned(),
@@ -153,7 +154,7 @@ impl<'a, T: PluginCategory + ?Sized> PluginLoader<'a, T> {
         let mut last_error = libloading::Error::IncompatibleSize;
         for p in candidates.iter() {
             match unsafe { Library::new(p.as_str()) } {
-                Ok(lib) => return Ok(lib),
+                Ok(lib) => return Ok((lib, p.as_str())),
                 Err(e) => last_error = e,
             }
         }
@@ -167,9 +168,12 @@ impl<'a, T: PluginCategory + ?Sized> PluginLoader<'a, T> {
         &mut self,
         candidates: &[String],
     ) -> SudachiResult<<T as PluginCategory>::BoxType> {
-        let lib = Self::try_load_library_from(candidates)?;
+        let (lib, path) = Self::try_load_library_from(candidates)?;
         let load_fn: Symbol<fn() -> SudachiResult<<T as PluginCategory>::BoxType>> =
-            unsafe { lib.get(b"load_plugin")? };
+            unsafe { lib.get(b"load_plugin") }.map_err(|e| PluginError::Libloading {
+                source: e,
+                message: format!("no load_plugin symbol in {}", path),
+            })?;
         let plugin = load_fn();
         self.libraries.push(lib);
         plugin
@@ -179,9 +183,9 @@ impl<'a, T: PluginCategory + ?Sized> PluginLoader<'a, T> {
 fn extract_plugin_class(val: &Value) -> SudachiResult<&str> {
     let obj = match val {
         Value::Object(v) => v,
-        _ => {
+        o => {
             return Err(SudachiError::ConfigError(ConfigError::InvalidFormat(
-                "plugin config must be an object".to_owned(),
+                format!("plugin config must be an object, was {}", o),
             )));
         }
     };
diff --git a/sudachi/src/plugin/mod.rs b/sudachi/src/plugin/mod.rs
index 14f8939..d7f88a2 100644
--- a/sudachi/src/plugin/mod.rs
+++ b/sudachi/src/plugin/mod.rs
@@ -69,10 +69,12 @@ pub(crate) struct Plugins {
 impl Plugins {
     pub(crate) fn load(cfg: &Config, grammar: &Grammar) -> SudachiResult<Plugins> {
         let plugins = Plugins {
-            connect_cost: load_plugins_of(cfg, grammar)?,
-            input_text: load_plugins_of(cfg, grammar)?,
-            oov: load_plugins_of(cfg, grammar)?,
-            path_rewrite: load_plugins_of(cfg, grammar)?,
+            connect_cost: load_plugins_of(cfg, grammar)
+                .map_err(|e| e.with_context("connect_cost"))?,
+            input_text: load_plugins_of(cfg, grammar).map_err(|e| e.with_context("input_text"))?,
+            oov: load_plugins_of(cfg, grammar).map_err(|e| e.with_context("oov"))?,
+            path_rewrite: load_plugins_of(cfg, grammar)
+                .map_err(|e| e.with_context("path_rewrite"))?,
         };
         Ok(plugins)
     }


Test patch:
diff --git a/python/tests/test_morpheme.py b/python/tests/test_morpheme.py
index 243a634..6692811 100644
--- a/python/tests/test_morpheme.py
+++ b/python/tests/test_morpheme.py
@@ -148,6 +148,16 @@ class TestTokenizer(unittest.TestCase):
         self.assertEqual(ms_a[0].surface(), '東京')
         self.assertEqual(ms_a[1].surface(), '都')
 
+    def test_morpheme_str_repr(self):
+        ms = self.tokenizer_obj.tokenize('東京都', SplitMode.A)
+        self.assertEqual(2, ms.size())
+        self.assertEqual(str(ms), '東京 都')
+        self.assertEqual(repr(ms), '<MorphemeList[\n  <Morpheme(東京, 0:2, (0, 5))>,\n  <Morpheme(都, 2:3, (0, 9))>,\n]>')
+        self.assertEqual(str(ms[0]), '東京')
+        self.assertEqual(str(ms[1]), '都')
+        self.assertEqual(repr(ms[0]), '<Morpheme(東京, 0:2, (0, 5))>')
+        self.assertEqual(repr(ms[1]), '<Morpheme(都, 2:3, (0, 9))>')
+
 
 if __name__ == '__main__':
     unittest.main()
diff --git a/sudachi/tests/common/mod.rs b/sudachi/tests/common/mod.rs
index b2fdf1e..d69287d 100644
--- a/sudachi/tests/common/mod.rs
+++ b/sudachi/tests/common/mod.rs
@@ -25,7 +25,7 @@ use std::sync::Arc;
 
 use sudachi::analysis::stateful_tokenizer::StatefulTokenizer;
 use sudachi::analysis::stateless_tokenizer::StatelessTokenizer;
-use sudachi::config::Config;
+use sudachi::config::{Config, ConfigBuilder};
 use sudachi::dic::dictionary::JapaneseDictionary;
 use sudachi::dic::{grammar::Grammar, header::Header, lexicon::Lexicon, DictionaryLoader};
 use sudachi::prelude::*;
@@ -45,6 +45,8 @@ pub fn dictionary_bytes_from_path<P: AsRef<Path>>(dictionary_path: P) -> Sudachi
     Ok(dictionary_bytes)
 }
 
+pub const LEX_CSV: &[u8] = include_bytes!("../resources/lex.csv");
+
 lazy_static! {
     pub static ref TEST_CONFIG: Config = {
         let test_config_path = "tests/resources/sudachi.json";
@@ -117,6 +119,7 @@ pub struct TestTokenizerBuilder<'a> {
     pub user: Vec<&'a [u8]>,
     pub mode: Mode,
     pub debug: bool,
+    pub config: Option<&'a [u8]>,
 }
 
 #[allow(unused)]
@@ -136,6 +139,11 @@ impl<'a> TestTokenizerBuilder<'a> {
         self
     }
 
+    pub fn config(mut self, data: &'static [u8]) -> Self {
+        self.config = Some(data);
+        self
+    }
+
     pub fn build(self) -> TestStatefulTokenizer {
         let mut sys = DictBuilder::new_system();
         sys.read_conn(
@@ -167,7 +175,12 @@ impl<'a> TestTokenizerBuilder<'a> {
             }
         }
 
-        let dic = JapaneseDictionary::from_cfg_storage(&TEST_CONFIG, data).unwrap();
+        let config = match self.config {
+            None => TEST_CONFIG.clone(),
+            Some(data) => ConfigBuilder::from_bytes(data).unwrap().build(),
+        };
+
+        let dic = JapaneseDictionary::from_cfg_storage(&config, data).unwrap();
         let rcdic = Rc::new(dic);
 
         TestStatefulTokenizer {
@@ -199,6 +212,7 @@ impl TestStatefulTokenizer {
             conn: None,
             mode: Mode::C,
             debug: false,
+            config: None,
         }
     }
 
diff --git a/sudachi/tests/resources/char.def b/sudachi/tests/resources/char.def
index 078fcea..9518237 100644
--- a/sudachi/tests/resources/char.def
+++ b/sudachi/tests/resources/char.def
@@ -1,3 +1,6 @@
+DEFAULT         0 1 0  # DEFAULT is a mandatory category!
+ALPHA           1 1 0
+
 0x0030..0x0039 NUMERIC  #0-9
 0x0041..0x005A ALPHA    #A-Z
 0x0061..0x007A ALPHA    #a-z
@@ -18,3 +21,16 @@
 0xFF10..0xFF19 NUMERIC
 0xFF21..0xFF3A ALPHA
 0xFF41..0xFF5A ALPHA
+
+# Unicode combining symbols 2021/12/03
+# from https://en.wikipedia.org/wiki/Combining_character
+0x0300..0x036F ALL NOOOVBOW # Combining Diacritical Marks
+0x1AB0..0x1AFF ALL NOOOVBOW # Combining Diacritical Marks Extended
+0x1DC0..0x1DFF ALL NOOOVBOW # Combining Diacritical Marks Supplement
+0x20D0..0x20FF ALL NOOOVBOW # Combining Diacritical Marks for Symbols
+0xFE20..0xFE2F ALL NOOOVBOW # Combining Half Marks
+0xFE00..0xFE0F ALL NOOOVBOW # https://codepoints.net/variation_selectors
+0x1F3FB..0x1F3FE ALL NOOOVBOW # emoji skin tone modifiers https://codepoints.net/U+1F3FF
+
+# Combination marks
+0x200C..0x200D ALL NOOOVBOW2 # Zero Width Non-Joiner/Joiner
\ No newline at end of file
diff --git a/sudachi/tests/resources/lex.csv b/sudachi/tests/resources/lex.csv
new file mode 100644
index 0000000..5b8ff32
--- /dev/null
+++ b/sudachi/tests/resources/lex.csv
@@ -0,0 +1,46 @@
+た,1,1,8729,た,助動詞,*,*,*,助動詞-タ,終止形-一般,タ,た,*,A,*,*,*,*
+に,2,2,11406,に,助詞,接続助詞,*,*,*,*,ニ,に,*,A,*,*,*,*
+に,3,3,4481,に,助詞,格助詞,*,*,*,*,ニ,に,*,A,*,*,*,*
+京都,6,6,5293,京都,名詞,固有名詞,地名,一般,*,*,キョウト,京都,*,A,*,*,*,1/5
+東,7,7,4675,東,名詞,普通名詞,一般,*,*,*,ヒガシ,東,*,A,*,*,*,*
+東京,6,6,2816,東京,名詞,固有名詞,地名,一般,*,*,トウキョウ,東京,*,A,*,*,*,*
+東京都,6,8,5320,東京都,名詞,固有名詞,地名,一般,*,*,トウキョウト,東京都,*,B,5/9,*,5/9,*
+行く,4,4,5105,行く,動詞,非自立可能,*,*,五段-カ行,終止形-一般,イク,行く,*,A,*,*,*,*
+行っ,5,5,5122,行っ,動詞,非自立可能,*,*,五段-カ行,連用形-促音便,イッ,行く,7,A,*,*,*,*
+都,8,8,2914,都,名詞,普通名詞,一般,*,*,*,ト,都,*,A,*,*,*,*
+アイ,7,7,4675,アイ,名詞,普通名詞,一般,*,*,*,アイ,アイ,*,A,*,*,*,*
+アイウ,7,7,4675,アイウ,名詞,普通名詞,一般,*,*,*,アイウ,アイウ,*,A,*,*,*,*
+アイアイウ,6,6,32766,アイウ,名詞,固有名詞,地名,一般,*,*,アイアイウ,アイアイウ,*,A,*,*,*,*
+0,9,9,2478,0,名詞,数詞,*,*,*,*,ゼロ,0,*,A,*,*,*,*
+1,9,9,2478,1,名詞,数詞,*,*,*,*,イチ,1,*,A,*,*,*,*
+2,9,9,2478,2,名詞,数詞,*,*,*,*,ニ,2,*,A,*,*,*,*
+3,9,9,2478,3,名詞,数詞,*,*,*,*,サン,3,*,A,*,*,*,*
+4,9,9,2478,4,名詞,数詞,*,*,*,*,ヨン,4,*,A,*,*,*,*
+5,9,9,2478,5,名詞,数詞,*,*,*,*,ゴ,5,*,A,*,*,*,*
+6,9,9,2478,6,名詞,数詞,*,*,*,*,ロク,6,*,A,*,*,*,*
+7,9,9,2478,7,名詞,数詞,*,*,*,*,ナナ,7,*,A,*,*,*,*
+8,9,9,2478,8,名詞,数詞,*,*,*,*,ハチ,8,*,A,*,*,*,*
+9,9,9,2478,9,名詞,数詞,*,*,*,*,キュウ,9,*,A,*,*,*,*
+〇,9,9,2478,〇,名詞,数詞,*,*,*,*,ゼロ,〇,*,A,*,*,*,*
+一,9,9,2478,一,名詞,数詞,*,*,*,*,イチ,一,*,A,*,*,*,*
+二,9,9,2478,二,名詞,数詞,*,*,*,*,ニ,二,*,A,*,*,*,*
+三,9,9,2478,三,名詞,数詞,*,*,*,*,サン,三,*,A,*,*,*,*
+四,9,9,2478,四,名詞,数詞,*,*,*,*,ヨン,四,*,A,*,*,*,*
+五,9,9,2478,五,名詞,数詞,*,*,*,*,ゴ,五,*,A,*,*,*,*
+六,9,9,2478,六,名詞,数詞,*,*,*,*,ロク,六,*,A,*,*,*,*
+七,9,9,2478,七,名詞,数詞,*,*,*,*,ナナ,七,*,A,*,*,*,*
+八,9,9,2478,八,名詞,数詞,*,*,*,*,ハチ,八,*,A,*,*,*,*
+九,9,9,2478,九,名詞,数詞,*,*,*,*,キュウ,九,*,A,*,*,*,*
+六三四,6,6,0,六三四,名詞,固有名詞,地名,一般,*,*,ムサシ,六三四,*,A,*,*,*,*
+いく,4,4,5105,いく,動詞,非自立可能,*,*,五段-カ行,終止形-一般,イク,行く,*,A,*,*,*,*
+いっ,5,5,5122,いっ,動詞,非自立可能,*,*,五段-カ行,連用形-促音便,イッ,行く,34,A,*,*,*,*
+012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789,9,9,2478,012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789,名詞,数詞,*,*,*,*,ゼロイチニサンヨンゴロクナナハチキュウゼロイチニサンヨンゴロクナナハチキュウゼロイチニサンヨンゴロクナナハチキュウゼロイチニサンヨンゴロクナナハチキュウゼロイチニサンヨンゴロクナナハチキュウゼロイチニサンヨンゴロクナナハチキュウゼロイチニサンヨンゴロクナナハチキュウゼロイチニサンヨンゴロクナナハチキュウゼロイチニサンヨンゴロクナナハチキュウゼロイチニサンヨンゴロクナナハチキュウゼロイチニサンヨンゴロクナナハチキュウゼロイチニサンヨンゴロクナナハチキュウゼロイチニサンヨンゴロクナナハチキュウゼロイチニサンヨンゴロクナナハチキュウゼロイチニサンヨンゴロクナナハチキュウゼロイチニサンヨンゴロクナナハチキュウゼロイチニサンヨンゴロクナナハチキュウゼロイチニサンヨンゴロクナナハチキュウゼロイチニサンヨンゴロクナナハチキュウゼロイチニサンヨンゴロクナナハチキュウゼロイチニサンヨンゴロクナナハチキュウゼロイチニサンヨンゴロクナナハチキュウゼロイチニサンヨンゴロクナナハチキュウゼロイチニサンヨンゴロクナナハチキュウゼロイチニサンヨンゴロクナナハチキュウゼロイチニサンヨンゴロクナナハチキュウゼロイチニサンヨンゴロクナナハチキュウゼロイチニサンヨンゴロクナナハチキュウゼロイチニサンヨンゴロクナナハチキュウゼロイチニサンヨンゴロクナナハチキュウ,012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789,*,A,*,*,*,*
+特a,8,8,2914,特a,名詞,普通名詞,一般,*,*,*,トクエー,特a,*,A,*,*,*,*
+な。な,8,8,2914,な。な,名詞,普通名詞,一般,*,*,*,ナナ,な。な,*,A,*,*,*,*
+ ,8,8,-20000, ,空白,*,*,*,*,*, , ,*,A,*,*,*,*
+X,8,8,-20000,X,補助記号,一般,*,*,*,*,X,X,*,A,*,*,*,*
+X,8,8,-20000,X,感動詞,一般,*,*,*,*,X,X,*,A,*,*,*,*
+X,8,8,-20000,X,名詞,普通名詞,サ変可能,*,*,*,X,X,*,A,*,*,*,*
+X,8,8,-20000,X,名詞,固有名詞,一般,*,*,*,X,X,*,A,*,*,*,*
+X,8,8,-20000,X,名詞,固有名詞,人名,一般,*,*,X,X,*,A,*,*,*,*
+X,8,8,-20000,X,名詞,固有名詞,地名,一般,*,*,X,X,*,A,*,*,*,*
\ No newline at end of file
diff --git a/sudachi/tests/resources/sudachi.oov.json b/sudachi/tests/resources/sudachi.oov.json
new file mode 100644
index 0000000..5512654
--- /dev/null
+++ b/sudachi/tests/resources/sudachi.oov.json
@@ -0,0 +1,27 @@
+{
+    "resourcePath" : "tests/resources/",
+    "systemDict" : "system.dic.test",
+    "userDict" : [ "user.dic.test" ],
+    "characterDefinitionFile" : "char.def",
+    "inputTextPlugin" : [
+        { "class" : "com.worksap.nlp.sudachi.DefaultInputTextPlugin" }
+    ],
+    "oovProviderPlugin" : [
+        { "class" : "com.worksap.nlp.sudachi.MeCabOovPlugin",
+          "charDef" : "char.def",
+          "unkDef" : "unk2.def" },
+        { "class" : "com.worksap.nlp.sudachi.SimpleOovPlugin",
+          "oovPOS" : [ "名詞", "普通名詞", "一般", "*", "*", "*" ],
+          "leftId" : 8,
+          "rightId" : 8,
+          "cost" : 6000 }
+    ],
+    "pathRewritePlugin" : [
+        { "class" : "com.worksap.nlp.sudachi.JoinNumericPlugin",
+          "enableNormalize" : true },
+        { "class" : "com.worksap.nlp.sudachi.JoinKatakanaOovPlugin",
+          "oovPOS" : [ "名詞", "普通名詞", "一般", "*", "*", "*" ],
+          "minLength" : 3
+        }
+    ]
+}
diff --git a/sudachi/tests/resources/unk2.def b/sudachi/tests/resources/unk2.def
new file mode 100644
index 0000000..17f3127
--- /dev/null
+++ b/sudachi/tests/resources/unk2.def
@@ -0,0 +1,2 @@
+DEFAULT,7,7,3857,補助記号,一般,*,*,*,*
+ALPHA,7,7,11633,名詞,普通名詞,一般,*,*,*
\ No newline at end of file
diff --git a/sudachi/tests/stateful_tokenizer.rs b/sudachi/tests/stateful_tokenizer.rs
index 482eb5f..b8541b6 100644
--- a/sudachi/tests/stateful_tokenizer.rs
+++ b/sudachi/tests/stateful_tokenizer.rs
@@ -21,7 +21,7 @@ use std::ops::Deref;
 use sudachi::prelude::Mode;
 
 mod common;
-use crate::common::TestStatefulTokenizer as TestTokenizer;
+use crate::common::{TestStatefulTokenizer as TestTokenizer, LEX_CSV};
 
 #[test]
 fn empty() {
@@ -143,3 +143,21 @@ fn split_middle() {
     assert_eq!(ms_a.get(1).begin(), 12);
     assert_eq!(ms_a.get(1).end(), 15);
 }
+
+const OOV_CFG: &[u8] = include_bytes!("resources/sudachi.oov.json");
+
+#[test]
+fn istanbul_is_not_splitted() {
+    let mut tok = TestTokenizer::builder(LEX_CSV).config(OOV_CFG).build();
+    let ms = tok.tokenize("İstanbul");
+    assert_eq!(ms.len(), 1);
+}
+
+#[test]
+fn emoji_are_not_splitted() {
+    let mut tok = TestTokenizer::builder(LEX_CSV).config(OOV_CFG).build();
+    assert_eq!(tok.tokenize("⏸").len(), 1);
+    assert_eq!(tok.tokenize("🦹‍♂️").len(), 1);
+    assert_eq!(tok.tokenize("🎅🏾").len(), 1);
+    assert_eq!(tok.tokenize("👳🏽‍♂").len(), 1);
+}