Repository: WorksApplications/sudachi.rs (language: python)

Issue:
Add an ability to convert result of surface() method to normalized variants by specifying a projection
Projection can be normalization transforms as specified by ChiTra or a user-passed callable.

Original surface will always be available as `raw_surface()`.

# Specification

* Add an `projection` argument to `Dictionary.create` which can accept either a string or a callable.
* When `projection` is passed, instead of usual `Morpheme.surface()` result, Sudachi.rs will produce a projected string.
* Projected strings are not cached, projection is computed each time on call to `surface()`
* Original string is available as `raw_surface()` method on `Morpheme`
Add an ability to convert result of surface() method to normalized variants by specifying a projection
Projection can be normalization transforms as specified by ChiTra or a user-passed callable.

Original surface will always be available as `raw_surface()`.

# Specification

* Add an `projection` argument to `Dictionary.create` which can accept either a string or a callable.
* When `projection` is passed, instead of usual `Morpheme.surface()` result, Sudachi.rs will produce a projected string.
* Projected strings are not cached, projection is computed each time on call to `surface()`
* Original string is available as `raw_surface()` method on `Morpheme`


Discussion / hints:
@eiennohito We need to consider the way to pass the projection type without modifying BertJapaneseTokenizer in transformers repo.

- sudachi_config_path
- sudachi_resource_dir
- sudachi_dict_type

These three arguments are the candidates for specifying projection type.

https://github.com/huggingface/transformers/blob/v4.29.0/src/transformers/models/bert_japanese/tokenization_bert_japanese.py#L533-L546
https://github.com/huggingface/transformers/blob/v4.29.0/src/transformers/models/bert_japanese/tokenization_bert_japanese.py#L533-L546
https://github.com/huggingface/transformers/blob/v4.29.0/src/transformers/models/bert_japanese/tokenization_bert_japanese.py#L590-L592
In the long term it would be better to submit a patch to huggingface transformers to provide `**kwargs` with options to all Japanese tokenizers though
@eiennohito We need to consider the way to pass the projection type without modifying BertJapaneseTokenizer in transformers repo.

- sudachi_config_path
- sudachi_resource_dir
- sudachi_dict_type

These three arguments are the candidates for specifying projection type.

https://github.com/huggingface/transformers/blob/v4.29.0/src/transformers/models/bert_japanese/tokenization_bert_japanese.py#L533-L546
https://github.com/huggingface/transformers/blob/v4.29.0/src/transformers/models/bert_japanese/tokenization_bert_japanese.py#L533-L546
https://github.com/huggingface/transformers/blob/v4.29.0/src/transformers/models/bert_japanese/tokenization_bert_japanese.py#L590-L592
In the long term it would be better to submit a patch to huggingface transformers to provide `**kwargs` with options to all Japanese tokenizers though

Fix patch:
diff --git a/python/py_src/sudachipy/config.py b/python/py_src/sudachipy/config.py
new file mode 100644
index 00000000..cf3b6b8c
--- /dev/null
+++ b/python/py_src/sudachipy/config.py
@@ -0,0 +1,79 @@
+#   Copyright (c) 2023 Works Applications Co., Ltd.
+#
+#   Licensed under the Apache License, Version 2.0 (the "License");
+#   you may not use this file except in compliance with the License.
+#   You may obtain a copy of the License at
+#
+#       http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#   distributed under the License is distributed on an "AS IS" BASIS,
+#   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#   See the License for the specific language governing permissions and
+#   limitations under the License.
+import dataclasses
+from dataclasses import dataclass, asdict
+from json import dumps
+
+@dataclass
+class Config:
+    """
+    SudachiPy rich configuration object.
+
+    Fields passed here will override the fields in the default configuration.
+    """
+    system: str = None
+    """
+    Path to dictionary or one of three strings: 'small', 'core', 'notcore'.    
+    If the file with the specified path does not exist and is not one of three special values, raise an error.
+    If you want to use dictionary with one of predefined names, use relative paths e.g. './core' instead of 'core'.
+    
+    If the value is one of three special values and there does not exist a file with the same name,
+    we try to load the dictionary from the SudachiDict_{system} installed package.
+    For example, for "core" we will try to load the dictionary from the installed SudachiDict_core package.
+    """
+    user: list[str] = None
+    """
+    Paths to user dictionaries, maximum number of user dictionaries is 14
+    """
+    projection: str = "surface"
+    """
+    Output the following field as the result of [Morpheme.surface()] instead of its value.
+    This option works for pre-tokenizers created for a given dictionary as well.
+    The original value is available as [Morpheme.raw_surface()].
+    
+    This option is created for chiTra integration.
+    
+    Available options:
+    - surface
+    - normalized
+    - reading
+    - dictionary 
+    - dictionary_and_surface 
+    - normalized_and_surface
+    - normalized_nouns
+    """
+
+    connectionCostPlugin: list = None
+    oovProviderPlugin: list = None
+    pathRewritePlugin: list = None
+    inputTextPlugin: list = None
+    characterDefinitionFile: str = None
+
+    def as_jsons(self):
+        """
+        Convert this Config object to the json string
+        """
+        return dumps(_filter_nulls(asdict(self)))
+
+    def update(self, **changes):
+        return dataclasses.replace(self, **changes)
+
+
+def _filter_nulls(data: dict) -> dict:
+    keys = list(data.keys())
+    for key in keys:
+        v = data[key]
+        if v is None:
+            del data[key]
+    return data
diff --git a/python/py_src/sudachipy/errors.py b/python/py_src/sudachipy/errors.py
new file mode 100644
index 00000000..e75e21cd
--- /dev/null
+++ b/python/py_src/sudachipy/errors.py
@@ -0,0 +1,16 @@
+#   Copyright (c) 2023 Works Applications Co., Ltd.
+#
+#   Licensed under the Apache License, Version 2.0 (the "License");
+#   you may not use this file except in compliance with the License.
+#   You may obtain a copy of the License at
+#
+#       http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#   distributed under the License is distributed on an "AS IS" BASIS,
+#   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#   See the License for the specific language governing permissions and
+#   limitations under the License.
+
+class SudachiError(Exception):
+    pass
\ No newline at end of file
diff --git a/python/py_src/sudachipy/sudachipy.pyi b/python/py_src/sudachipy/sudachipy.pyi
index bb7bce59..78230aa9 100644
--- a/python/py_src/sudachipy/sudachipy.pyi
+++ b/python/py_src/sudachipy/sudachipy.pyi
@@ -1,4 +1,5 @@
 from typing import ClassVar, Iterator, List, Tuple, Union, Callable, Iterable, Optional, Literal, Set
+from sudachipy.config import Config
 
 POS = Tuple[str, str, str, str, str, str]
 # POS element
@@ -40,16 +41,18 @@ class Dictionary:
     """
 
     @classmethod
-    def __init__(cls, config_path: Optional[str] = ..., resource_dir: Optional[str] = ..., dict: Optional[str] = None, *, dict_type: Optional[str] = None) -> None:
+    def __init__(cls, config_path: Optional[str | Config] = ..., resource_dir: Optional[str] = ..., dict: Optional[str] = None,
+                 dict_type: Optional[str] = None, *, config: Optional[str | Config] = ...) -> None:
         """
         Creates a sudachi dictionary.
 
-        If both config.systemDict and dict_type are not given, `sudachidict_core` is used.
-        If both config.systemDict and dict_type are given, dict_type is used.
+        If both config.systemDict and dict are not given, `sudachidict_core` is used.
+        If both config.systemDict and dict are given, dict_type is used.
 
-        :param config_path: path to the configuration JSON file
+        :param config_path: path to the configuration JSON file, config json as a string, or a [sudachipy.config.Config] object
+        :param config: alias to config_path, only one of them can be specified at the same time
         :param resource_dir: path to the resource directory folder
-        :param dict: type of pre-packaged dictionary, referring to sudachidict_<dict> packages on PyPI: https://pypi.org/search/?q=sudachidict.
+        :param dict: type of pre-packaged system dictionary, referring to sudachidict_<dict> packages on PyPI: https://pypi.org/search/?q=sudachidict.
             Also, can be an _absolute_ path to a compiled dictionary file.
         :param dict_type: deprecated alias to dict
         """
@@ -63,13 +66,16 @@ class Dictionary:
 
     def create(self,
                mode: SplitMode = SplitMode.C,
-               fields: FieldSet = None) -> Tokenizer:
+               fields: FieldSet = None,
+               *,
+               projection: str = None) -> Tokenizer:
         """
         Creates a Sudachi Tokenizer.
 
         :param mode: sets the analysis mode for this Tokenizer
         :param fields: load only a subset of fields.
             See https://worksapplications.github.io/sudachi.rs/python/topics/subsetting.html
+        :param projection: Projection override for created Tokenizer. See Config.projection for values.
         """
         ...
 
@@ -92,7 +98,9 @@ class Dictionary:
     def pre_tokenizer(self,
                       mode: SplitMode = SplitMode.C,
                       fields: FieldSet = None,
-                      handler: Optional[Callable[[int, object, MorphemeList], list]] = None) -> object:
+                      handler: Optional[Callable[[int, object, MorphemeList], list]] = None,
+                      *,
+                      projection: str = None) -> object:
         """
         Creates HuggingFace Tokenizers-compatible PreTokenizer.
         Requires package `tokenizers` to be installed.
@@ -102,6 +110,7 @@ class Dictionary:
         :param handler: custom callable to transform MorphemeList into list of tokens. See https://github.com/huggingface/tokenizers/blob/master/bindings/python/examples/custom_components.py
         First two parameters are the index (int) and HuggingFace NormalizedString.
         The handler must return a List[NormalizedString]. By default, just segment the tokens.
+        :param projection: Projection override for created Tokenizer. See Config.projection for values.
         """
         ...
 
@@ -198,7 +207,14 @@ class Morpheme:
 
     def surface(self) -> str:
         """
-        Returns the surface.
+        Returns the projected string for the given morpheme (by default surface).
+        See Config.projection
+        """
+        ...
+
+    def raw_surface(self) -> str:
+        """
+        Returns the surface string no matter the value of Config.projection
         """
         ...
 
diff --git a/python/src/build.rs b/python/src/build.rs
index d6ff4ef8..a6005b26 100644
--- a/python/src/build.rs
+++ b/python/src/build.rs
@@ -17,7 +17,6 @@
 use crate::dictionary::get_default_resource_dir;
 use crate::errors;
 use pyo3::prelude::*;
-use pyo3::type_object::PyTypeInfo;
 use pyo3::types::{PyBytes, PyList, PyString, PyTuple, PyType};
 use std::fs::{File, OpenOptions};
 use std::io::BufWriter;
@@ -136,10 +135,10 @@ fn as_data_source<'p>(py: Python<'p>, data: &'p PyAny) -> PyResult<DataSource<'p
     if data.is_instance(path)? {
         let pypath = data.call_method0("resolve")?.str()?;
         Ok(DataSource::File(Path::new(pypath.to_str()?)))
-    } else if data.is_instance(PyString::type_object(py))? {
+    } else if data.is_instance_of::<PyString>() {
         let pypath = data.str()?;
         Ok(DataSource::File(Path::new(pypath.to_str()?)))
-    } else if data.is_instance(PyBytes::type_object(py))? {
+    } else if data.is_instance_of::<PyBytes>() {
         let data = data.downcast::<PyBytes>()?;
         Ok(DataSource::Data(data.as_bytes()))
     } else {
diff --git a/python/src/dictionary.rs b/python/src/dictionary.rs
index 19f52ac4..03fb7ad2 100644
--- a/python/src/dictionary.rs
+++ b/python/src/dictionary.rs
@@ -1,5 +1,5 @@
 /*
- *  Copyright (c) 2021 Works Applications Co., Ltd.
+ *  Copyright (c) 2021-2023 Works Applications Co., Ltd.
  *
  *  Licensed under the Apache License, Version 2.0 (the "License");
  *  you may not use this file except in compliance with the License.
@@ -14,16 +14,19 @@
  *  limitations under the License.
  */
 
+use std::convert::TryFrom;
 use std::fmt::Write;
 use std::path::{Path, PathBuf};
 use std::sync::Arc;
 
 use pyo3::exceptions::PyException;
+use pyo3::methods::OkWrap;
 use pyo3::prelude::*;
 use pyo3::types::{PySet, PyString, PyTuple};
 
+use crate::errors::{wrap, wrap_ctx, SudachiError as SudachiErr};
 use sudachi::analysis::stateless_tokenizer::DictionaryAccess;
-use sudachi::config::{Config, ConfigBuilder};
+use sudachi::config::{Config, ConfigBuilder, SurfaceProjection};
 use sudachi::dic::dictionary::JapaneseDictionary;
 use sudachi::dic::grammar::Grammar;
 use sudachi::dic::lexicon_set::LexiconSet;
@@ -32,14 +35,19 @@ use sudachi::plugin::input_text::InputTextPlugin;
 use sudachi::plugin::oov::OovProviderPlugin;
 use sudachi::plugin::path_rewrite::PathRewritePlugin;
 
-use crate::morpheme::PyMorphemeListWrapper;
+use crate::morpheme::{PyMorphemeListWrapper, PyProjector};
 use crate::pos_matcher::PyPosMatcher;
 use crate::pretokenizer::PyPretokenizer;
+use crate::projection::{morpheme_projection, parse_projection_opt, resolve_projection};
 use crate::tokenizer::{PySplitMode, PyTokenizer};
 
 pub(crate) struct PyDicData {
     pub(crate) dictionary: JapaneseDictionary,
     pub(crate) pos: Vec<Py<PyTuple>>,
+    /// Compute default string representation for a morpheme using vtable dispatch.
+    /// None by default (if outputting surface as it is)
+    /// This is default per-dictionary value, can be overriden when creating tokenizers and pre-tokenizers
+    pub(crate) projection: PyProjector,
 }
 
 impl DictionaryAccess for PyDicData {
@@ -92,32 +100,34 @@ impl PyDictionary {
     ///     Also, can be an _absolute_ path to a compiled dictionary file.
     /// :param dict_type: deprecated alias to dict
     #[new]
-    #[pyo3(signature=(config_path = None, resource_dir = None, dict = None, dict_type = None))]
+    #[pyo3(signature=(config_path = None, resource_dir = None, dict = None, dict_type = None, *, config = None))]
     fn new(
         py: Python,
-        config_path: Option<PathBuf>,
+        config_path: Option<&PyAny>,
         resource_dir: Option<PathBuf>,
         dict: Option<&str>,
         dict_type: Option<&str>,
+        config: Option<&PyAny>,
     ) -> PyResult<Self> {
-        let config_path = match config_path {
-            None => Some(get_default_setting_path(py)?),
-            Some(v) => Some(v),
+        if config.is_some() && config_path.is_some() {
+            return Err(SudachiErr::new_err("Both config and config_path options were specified at the same time, use one of them"));
+        }
+
+        let default_config = read_default_config(py)?;
+
+        let config_builder = match config.or(config_path) {
+            None => default_config,
+            Some(v) => read_config(v)?.fallback(&default_config),
         };
+
         let resource_dir = match resource_dir {
             None => Some(get_default_resource_dir(py)?),
             Some(v) => Some(v),
         };
+
         let dict_path = match dict.or(dict_type) {
             None => None,
-            Some(dt) => {
-                let path = Path::new(dt);
-                if path.is_absolute() && path.exists() {
-                    Some(path.to_path_buf())
-                } else {
-                    Some(find_dict_path(py, dt)?)
-                }
-            }
+            Some(dt) => Some(locate_system_dict(py, Path::new(dt))?),
         };
 
         if dict_type.is_some() {
@@ -130,10 +140,6 @@ impl PyDictionary {
             )?;
         }
 
-        let config_builder = ConfigBuilder::from_opt_file(config_path.as_deref()).map_err(|e| {
-            PyException::new_err(format!("Error loading config: {}", e.to_string()))
-        })?;
-
         let config_builder = match resource_dir {
             Some(p) => config_builder.resource_path(p),
             None => config_builder,
@@ -157,10 +163,22 @@ impl PyDictionary {
                 system_dict.display()
             );
             config.system_dict = Some(system_dict);
+        } else {
+            // resolve system dictionary alias to full path
+            let system_dict = config.system_dict.as_deref().unwrap();
+            if let Some(kind @ ("small" | "core" | "full")) = system_dict.to_str() {
+                let system_dict = find_dict_path(py, kind)?;
+                assert!(
+                    system_dict.exists(),
+                    "system dictionary {} did not exist",
+                    system_dict.display()
+                );
+                config.system_dict = Some(system_dict)
+            }
         }
 
         let jdic = JapaneseDictionary::from_cfg(&config).map_err(|e| {
-            PyException::new_err(format!(
+            SudachiErr::new_err(format!(
                 "Error while constructing dictionary: {}",
                 e.to_string()
             ))
@@ -176,9 +194,16 @@ impl PyDictionary {
             })
             .collect();
 
+        let projection = if config.projection == SurfaceProjection::Surface {
+            None
+        } else {
+            Some(morpheme_projection(config.projection, &jdic))
+        };
+
         let dic_data = PyDicData {
             dictionary: jdic,
             pos: pos_data,
+            projection,
         };
 
         let dictionary = Arc::new(dic_data);
@@ -196,12 +221,29 @@ impl PyDictionary {
     ///     See https://worksapplications.github.io/sudachi.rs/python/topics/subsetting.html
     #[pyo3(
         text_signature = "($self, mode: sudachipy.SplitMode = sudachipy.SplitMode.C) -> sudachipy.Tokenizer",
-        signature = (mode = None, fields = None)
+        signature = (mode = None, fields = None, *, projection = None)
     )]
-    fn create(&self, mode: Option<PySplitMode>, fields: Option<&PySet>) -> PyResult<PyTokenizer> {
+    fn create(
+        &self,
+        mode: Option<PySplitMode>,
+        fields: Option<&PySet>,
+        projection: Option<&PyString>,
+    ) -> PyResult<PyTokenizer> {
         let mode = mode.unwrap_or(PySplitMode::C).into();
         let fields = parse_field_subset(fields)?;
-        let tok = PyTokenizer::new(self.dictionary.as_ref().unwrap().clone(), mode, fields);
+        let mut required_fields = self.config.projection.required_subset();
+        let dict = self.dictionary.as_ref().unwrap().clone();
+        let projobj = if let Some(s) = projection {
+            let proj = wrap(SurfaceProjection::try_from(s.to_str()?))?;
+            required_fields = proj.required_subset();
+            Some(morpheme_projection(proj, &dict))
+        } else {
+            None
+        };
+
+        let projobj = resolve_projection(projobj, &dict.projection);
+
+        let tok = PyTokenizer::new(dict, mode, fields | required_fields, projobj);
         Ok(tok)
     }
 
@@ -235,7 +277,7 @@ impl PyDictionary {
     /// :type fields: Set[str]
     #[pyo3(
         text_signature = "($self, mode, fields, handler) -> tokenizers.PreTokenizer)",
-        signature = (mode = None, fields = None, handler = None)
+        signature = (mode = None, fields = None, handler = None, *, projection = None)
     )]
     fn pre_tokenizer<'p>(
         &'p self,
@@ -243,28 +285,34 @@ impl PyDictionary {
         mode: Option<PySplitMode>,
         fields: Option<&PySet>,
         handler: Option<Py<PyAny>>,
+        projection: Option<&PyString>,
     ) -> PyResult<&'p PyAny> {
         let mode = mode.unwrap_or(PySplitMode::C).into();
         let subset = parse_field_subset(fields)?;
         if let Some(h) = handler.as_ref() {
             if !h.as_ref(py).is_callable() {
-                return Err(PyException::new_err("handler must be callable"));
+                return Err(SudachiErr::new_err("handler must be callable"));
             }
         }
 
-        // we don't need any fields when handler is not present
-        let subset = if handler.is_none() {
-            InfoSubset::empty()
+        let dict = self.dictionary.as_ref().unwrap().clone();
+
+        let mut required_fields = if handler.is_none() {
+            self.config.projection.required_subset()
+        } else {
+            subset | self.config.projection.required_subset()
+        };
+
+        let passed_projector = if let Some(s) = projection {
+            let proj = wrap(SurfaceProjection::try_from(s.to_str()?))?;
+            required_fields = proj.required_subset();
+            Some(morpheme_projection(proj, &dict))
         } else {
-            subset
+            None
         };
 
-        let internal = PyPretokenizer::new(
-            self.dictionary.as_ref().unwrap().clone(),
-            mode,
-            subset,
-            handler,
-        );
+        let projector = resolve_projection(passed_projector, &dict.projection);
+        let internal = PyPretokenizer::new(dict, mode, subset, handler, projector);
         let internal_cell = PyCell::new(py, internal)?;
         let module = py.import("tokenizers.pre_tokenizers")?;
         module
@@ -302,16 +350,12 @@ impl PyDictionary {
         // this needs to be a variable
         let mut borrow = l.try_borrow_mut();
         let out_list = match borrow {
-            Err(_) => return Err(PyException::new_err("out was used twice at the same time")),
+            Err(_) => return Err(SudachiErr::new_err("out was used twice at the same time")),
             Ok(ref mut ms) => ms.internal_mut(py),
         };
 
         out_list.clear();
-
-        out_list.lookup(surface, InfoSubset::all()).map_err(|e| {
-            PyException::new_err(format!("Failed to lookup words for {}: {:?}", surface, e))
-        })?;
-
+        wrap_ctx(out_list.lookup(surface, InfoSubset::all()), surface)?;
         Ok(l)
     }
 
@@ -329,21 +373,16 @@ impl PyDictionary {
     }
 
     fn __repr__(&self) -> PyResult<String> {
-        config_repr(&self.config).map_err(|e| {
-            return PyException::new_err(format!("{:?}", e));
-        })
+        wrap(config_repr(&self.config))
     }
 }
 
 fn config_repr(cfg: &Config) -> Result<String, std::fmt::Error> {
     let mut result = String::from("<SudachiDictionary(");
-    write!(
-        result,
-        "system={}",
-        cfg.resolved_system_dict()
-            .map(|p| p.to_string_lossy().into_owned())
-            .unwrap_or_else(|e| e.to_string())
-    )?;
+    match cfg.resolved_system_dict() {
+        Ok(path) => write!(result, "system={}", path.display()),
+        Err(e) => write!(result, "system=<err:{}>", e),
+    }?;
     write!(result, ", user=[")?;
     match cfg.resolved_user_dicts() {
         Ok(dicts) => {
@@ -357,7 +396,7 @@ fn config_repr(cfg: &Config) -> Result<String, std::fmt::Error> {
             }
         }
         Err(e) => {
-            write!(result, "{:?}", e)?;
+            write!(result, "<err:{:?}>", e)?;
         }
     }
 
@@ -365,10 +404,44 @@ fn config_repr(cfg: &Config) -> Result<String, std::fmt::Error> {
     Ok(result)
 }
 
-pub(crate) fn get_default_setting_path(py: Python) -> PyResult<PathBuf> {
+fn read_config_from_fs(path: Option<&Path>) -> PyResult<ConfigBuilder> {
+    wrap(ConfigBuilder::from_opt_file(path))
+}
+
+fn read_config(config_opt: &PyAny) -> PyResult<ConfigBuilder> {
+    if config_opt.is_instance_of::<PyString>() {
+        let config_str = config_opt.str()?.to_str()?.trim();
+        // looks like json
+        if config_str.starts_with("{") && config_str.ends_with("}") {
+            let result = ConfigBuilder::from_bytes(config_str.as_bytes());
+            return wrap(result);
+        }
+        let p = Path::new(config_str);
+        if p.exists() && p.is_file() {
+            return read_config_from_fs(Some(p));
+        }
+        return Err(SudachiErr::new_err(format!(
+            "config file [{}] do not exist or is not a file",
+            p.display()
+        )));
+    }
+    let py = config_opt.py();
+    let cfg_type = py.import("sudachipy.config")?.getattr("Config")?;
+    if config_opt.is_instance(cfg_type)? {
+        let cfg_as_str = config_opt.call_method0("as_jsons")?;
+        return read_config(cfg_as_str);
+    }
+    Err(SudachiErr::new_err((
+        format!("passed config was not a string, json object or sudachipy.config.Config object"),
+        config_opt.wrap(py)?,
+    )))
+}
+
+pub(crate) fn read_default_config(py: Python) -> PyResult<ConfigBuilder> {
     let path = PyModule::import(py, "sudachipy")?.getattr("_DEFAULT_SETTINGFILE")?;
     let path = path.downcast::<PyString>()?.to_str()?;
-    Ok(PathBuf::from(path))
+    let path = PathBuf::from(path);
+    wrap_ctx(ConfigBuilder::from_opt_file(Some(&path)), &path)
 }
 
 pub(crate) fn get_default_resource_dir(py: Python) -> PyResult<PathBuf> {
@@ -386,6 +459,19 @@ fn find_dict_path(py: Python, dict_type: &str) -> PyResult<PathBuf> {
     Ok(PathBuf::from(path))
 }
 
+fn locate_system_dict(py: Python, path: &Path) -> PyResult<PathBuf> {
+    if path.exists() && path.is_file() {
+        return Ok(path.to_owned());
+    }
+    match path.to_str() {
+        Some(name @ ("small" | "core" | "full")) => find_dict_path(py, name),
+        _ => Err(SudachiErr::new_err(format!(
+            "invalid dictionary path {:?}",
+            path
+        ))),
+    }
+}
+
 fn parse_field_subset(data: Option<&PySet>) -> PyResult<InfoSubset> {
     if data.is_none() {
         return Ok(InfoSubset::all());
diff --git a/python/src/errors.rs b/python/src/errors.rs
index 91317b28..04827fd4 100644
--- a/python/src/errors.rs
+++ b/python/src/errors.rs
@@ -14,20 +14,22 @@
  *  limitations under the License.
  */
 
-use pyo3::exceptions::PyException;
-use pyo3::PyResult;
+use pyo3::{import_exception, PyResult};
 use std::fmt::{Debug, Display};
 
+// Sudachi exception class is defined in Python
+import_exception!(sudachipy.errors, SudachiError);
+
 pub fn wrap<T, E: Display>(v: Result<T, E>) -> PyResult<T> {
     match v {
         Ok(v) => Ok(v),
-        Err(e) => Err(PyException::new_err(format!("{}", e))),
+        Err(e) => Err(SudachiError::new_err(format!("{}", e))),
     }
 }
 
 pub fn wrap_ctx<T, E: Display, C: Debug + ?Sized>(v: Result<T, E>, ctx: &C) -> PyResult<T> {
     match v {
         Ok(v) => Ok(v),
-        Err(e) => Err(PyException::new_err(format!("{:?}: {}", ctx, e))),
+        Err(e) => Err(SudachiError::new_err(format!("{:?}: {}", ctx, e))),
     }
 }
diff --git a/python/src/lib.rs b/python/src/lib.rs
index 1ed8a53b..68a9c91d 100644
--- a/python/src/lib.rs
+++ b/python/src/lib.rs
@@ -22,6 +22,7 @@ mod errors;
 mod morpheme;
 mod pos_matcher;
 mod pretokenizer;
+mod projection;
 mod tokenizer;
 mod word_info;
 
diff --git a/python/src/morpheme.rs b/python/src/morpheme.rs
index b3e4fba9..d4f744d2 100644
--- a/python/src/morpheme.rs
+++ b/python/src/morpheme.rs
@@ -25,17 +25,19 @@ use pyo3::types::{PyList, PyString, PyTuple, PyType};
 use sudachi::prelude::{Morpheme, MorphemeList};
 
 use crate::dictionary::{PyDicData, PyDictionary};
+use crate::projection::MorphemeProjection;
 use crate::tokenizer::PySplitMode;
 use crate::word_info::PyWordInfo;
 
-type PyMorphemeList = MorphemeList<Arc<PyDicData>>;
+pub(crate) type PyMorphemeList = MorphemeList<Arc<PyDicData>>;
+pub(crate) type PyProjector = Option<Arc<dyn MorphemeProjection + Send + Sync>>;
 
 /// A list of morphemes
 #[pyclass(module = "sudachipy.morphemelist", name = "MorphemeList")]
-#[repr(transparent)]
 pub struct PyMorphemeListWrapper {
     /// use `internal()` function instead
     inner: PyMorphemeList,
+    projection: PyProjector,
 }
 
 // PyMorphemeListWrapper is used only when GIL is active,
@@ -45,8 +47,24 @@ unsafe impl Send for PyMorphemeListWrapper {}
 
 impl PyMorphemeListWrapper {
     pub(crate) fn new(dict: Arc<PyDicData>) -> Self {
+        let proj = dict.projection.clone();
         Self {
             inner: PyMorphemeList::empty(dict),
+            projection: proj,
+        }
+    }
+
+    pub(crate) fn from_components(list: PyMorphemeList, projection: PyProjector) -> Self {
+        Self {
+            inner: list,
+            projection,
+        }
+    }
+
+    pub(crate) fn projection(&self) -> Option<&dyn MorphemeProjection> {
+        match &self.projection {
+            None => None,
+            Some(p) => Some(p.as_ref()),
         }
     }
 
@@ -56,6 +74,7 @@ impl PyMorphemeListWrapper {
     }
 
     /// Borrow internals immutable. GIL token proves access.
+    #[inline]
     pub(crate) fn internal(&self, _py: Python) -> &PyMorphemeList {
         &self.inner
     }
@@ -64,6 +83,7 @@ impl PyMorphemeListWrapper {
     pub(crate) fn empty_clone(&self, _py: Python) -> Self {
         Self {
             inner: self.inner.empty_clone(),
+            projection: self.projection.clone(),
         }
     }
 }
@@ -81,8 +101,11 @@ impl PyMorphemeListWrapper {
             1,
         )?;
 
+        let cloned = dict.dictionary.as_ref().unwrap().clone();
+        let proj = cloned.projection.clone();
         Ok(Self {
-            inner: PyMorphemeList::empty(dict.dictionary.as_ref().unwrap().clone()),
+            inner: PyMorphemeList::empty(cloned),
+            projection: proj,
         })
     }
 
@@ -175,14 +198,6 @@ impl PyMorphemeListWrapper {
     }
 }
 
-impl From<MorphemeList<Arc<PyDicData>>> for PyMorphemeListWrapper {
-    fn from(morpheme_list: MorphemeList<Arc<PyDicData>>) -> Self {
-        Self {
-            inner: morpheme_list,
-        }
-    }
-}
-
 /// A morpheme (basic semantic unit of language).
 #[pyclass(module = "sudachipy.morphemelist", name = "MorphemeIter")]
 pub struct PyMorphemeIter {
@@ -227,7 +242,7 @@ impl<'py> Deref for MorphemeRef<'py> {
     }
 }
 
-#[pyclass(module = "sudachipy.morpheme", name = "Morpheme")]
+#[pyclass(module = "sudachipy.morpheme", name = "Morpheme", frozen)]
 pub struct PyMorpheme {
     list: Py<PyMorphemeListWrapper>,
     index: usize,
@@ -278,8 +293,19 @@ impl PyMorpheme {
 
     /// Returns the substring of input text corresponding to the morpheme
     #[pyo3(text_signature = "($self) -> str")]
-    fn surface<'py>(&'py self, py: Python<'py>) -> PyObject {
-        self.morph(py).surface().deref().into_py(py)
+    fn surface<'py>(&'py self, py: Python<'py>) -> &'py PyString {
+        let list = self.list(py);
+        let morph = self.morph(py);
+        match list.projection() {
+            None => PyString::new(py, morph.surface().deref()),
+            Some(proj) => proj.project(morph.deref(), py),
+        }
+    }
+
+    /// Returns the substring of input text corresponding to the morpheme
+    #[pyo3(text_signature = "($self) -> str")]
+    fn raw_surface<'py>(&'py self, py: Python<'py>) -> &'py PyString {
+        PyString::new(py, self.morph(py).surface().deref())
     }
 
     /// Returns the part of speech as a six-element tuple.
@@ -418,7 +444,7 @@ impl PyMorpheme {
         m.end_c() - m.begin_c()
     }
 
-    pub fn __str__<'py>(&'py self, py: Python<'py>) -> PyObject {
+    pub fn __str__<'py>(&'py self, py: Python<'py>) -> &'py PyString {
         self.surface(py)
     }
 
diff --git a/python/src/pretokenizer.rs b/python/src/pretokenizer.rs
index 157d0fb6..755f040b 100644
--- a/python/src/pretokenizer.rs
+++ b/python/src/pretokenizer.rs
@@ -16,12 +16,15 @@
 
 use crate::dictionary::PyDicData;
 use crate::errors::wrap;
-use crate::morpheme::PyMorphemeListWrapper;
+use crate::morpheme::{PyMorphemeList, PyMorphemeListWrapper, PyProjector};
+use pyo3::intern;
 use pyo3::prelude::*;
-use pyo3::types::{PyList, PySlice, PyTuple};
+use pyo3::sync::GILOnceCell;
+use pyo3::types::{PyList, PySlice, PyTuple, PyType};
 use std::cell::RefCell;
 use std::sync::Arc;
 
+use crate::projection::MorphemeProjection;
 use sudachi::analysis::stateful_tokenizer::StatefulTokenizer;
 use sudachi::dic::subset::InfoSubset;
 use sudachi::prelude::Mode;
@@ -83,6 +86,7 @@ pub struct PyPretokenizer {
     subset: InfoSubset,
     tokenizers: ThreadLocal<RefCell<PerThreadPreTokenizer>>,
     handler: Option<Py<PyAny>>,
+    projection: PyProjector,
 }
 
 impl PyPretokenizer {
@@ -91,6 +95,7 @@ impl PyPretokenizer {
         mode: Mode,
         subset: InfoSubset,
         handler: Option<Py<PyAny>>,
+        projection: PyProjector,
     ) -> PyPretokenizer {
         Self {
             dict,
@@ -98,6 +103,7 @@ impl PyPretokenizer {
             subset,
             tokenizers: ThreadLocal::new(),
             handler,
+            projection,
         }
     }
 
@@ -135,17 +141,12 @@ impl PyPretokenizer {
         let morphs = cell.result();
         match self.handler.as_ref() {
             None => {
-                let result = PyList::empty(py);
                 let py_ref = morphs.borrow(py);
                 let morphs = py_ref.internal(py);
-                for idx in 0..morphs.len() {
-                    let node = morphs.get(idx);
-                    let slice = PySlice::new(py, node.begin_c() as isize, node.end_c() as isize, 1);
-                    let args = PyTuple::new(py, [slice]);
-                    let substring = string.call_method1("slice", args)?;
-                    result.append(substring)?;
+                match self.projection.as_deref() {
+                    None => make_result_for_surface(py, morphs, string),
+                    Some(p) => make_result_for_projection(py, morphs, p),
                 }
-                Ok(result)
             }
             Some(h) => {
                 let mrp: &PyAny = morphs.as_ref(py);
@@ -164,3 +165,43 @@ impl PyPretokenizer {
         data.call_method1("split", PyTuple::new(py, [self_]))
     }
 }
+
+fn make_result_for_surface<'py>(
+    py: Python<'py>,
+    morphs: &PyMorphemeList,
+    string: &'py PyAny,
+) -> PyResult<&'py PyAny> {
+    let result = PyList::empty(py);
+    for idx in 0..morphs.len() {
+        let node = morphs.get(idx);
+        let slice = PySlice::new(py, node.begin_c() as isize, node.end_c() as isize, 1);
+        let args = PyTuple::new(py, [slice]);
+        let substring = string.call_method1(intern!(py, "slice"), args)?;
+        result.append(substring)?;
+    }
+    Ok(result)
+}
+
+fn make_result_for_projection<'py>(
+    py: Python<'py>,
+    morphs: &PyMorphemeList,
+    proj: &dyn MorphemeProjection,
+) -> PyResult<&'py PyAny> {
+    let result = PyList::empty(py);
+    let nstring = {
+        static NORMALIZED_STRING: GILOnceCell<Py<PyType>> = pyo3::sync::GILOnceCell::new();
+        NORMALIZED_STRING.get_or_try_init(py, || -> PyResult<Py<PyType>> {
+            let ns = py.import("tokenizers")?.getattr("NormalizedString")?;
+            let tpe = ns.downcast::<PyType>();
+            tpe.map(|x| x.into_py(py)).map_err(|e| e.into())
+        })?
+    };
+    for idx in 0..morphs.len() {
+        let node = morphs.get(idx);
+        let value = proj.project(&node, py);
+        let args = PyTuple::new(py, [value]);
+        let substring = nstring.call1(py, args)?;
+        result.append(substring)?;
+    }
+    Ok(result)
+}
diff --git a/python/src/projection.rs b/python/src/projection.rs
new file mode 100644
index 00000000..c15bf540
--- /dev/null
+++ b/python/src/projection.rs
@@ -0,0 +1,199 @@
+/*
+ *  Copyright (c) 2023 Works Applications Co., Ltd.
+ *
+ *  Licensed under the Apache License, Version 2.0 (the "License");
+ *  you may not use this file except in compliance with the License.
+ *  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *   Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+
+use crate::dictionary::PyDicData;
+use crate::morpheme::PyProjector;
+use pyo3::types::PyString;
+use pyo3::{PyResult, Python};
+use std::convert::TryFrom;
+use std::ops::Deref;
+use std::sync::Arc;
+use sudachi::analysis::stateless_tokenizer::DictionaryAccess;
+use sudachi::config::SurfaceProjection;
+use sudachi::pos::PosMatcher;
+use sudachi::prelude::Morpheme;
+
+pub(crate) trait MorphemeProjection {
+    fn project<'py>(&self, m: &Morpheme<Arc<PyDicData>>, py: Python<'py>) -> &'py PyString;
+}
+
+struct Surface {}
+
+impl MorphemeProjection for Surface {
+    fn project<'py>(&self, m: &Morpheme<Arc<PyDicData>>, py: Python<'py>) -> &'py PyString {
+        PyString::new(py, m.surface().deref())
+    }
+}
+
+struct Mapped<F: for<'a> Fn(&'a Morpheme<'a, Arc<PyDicData>>) -> &'a str> {
+    func: F,
+}
+
+impl<F: for<'a> Fn(&'a Morpheme<'a, Arc<PyDicData>>) -> &'a str> MorphemeProjection for Mapped<F> {
+    fn project<'py>(&self, m: &Morpheme<Arc<PyDicData>>, py: Python<'py>) -> &'py PyString {
+        PyString::new(py, (self.func)(m))
+    }
+}
+
+struct DictionaryAndSurface {
+    matcher: PosMatcher,
+}
+
+impl DictionaryAndSurface {
+    fn new<D: DictionaryAccess>(dic: &D) -> Self {
+        let matcher = conjugating_matcher(dic);
+        DictionaryAndSurface { matcher }
+    }
+}
+
+impl MorphemeProjection for DictionaryAndSurface {
+    fn project<'py>(&self, m: &Morpheme<Arc<PyDicData>>, py: Python<'py>) -> &'py PyString {
+        if self.matcher.matches_id(m.part_of_speech_id()) {
+            PyString::new(py, m.surface().deref())
+        } else {
+            PyString::new(py, m.dictionary_form())
+        }
+    }
+}
+
+struct NormalizedAndSurface {
+    matcher: PosMatcher,
+}
+
+impl NormalizedAndSurface {
+    fn new<D: DictionaryAccess>(dic: &D) -> Self {
+        let matcher = conjugating_matcher(dic);
+        NormalizedAndSurface { matcher }
+    }
+}
+
+impl MorphemeProjection for NormalizedAndSurface {
+    fn project<'py>(&self, m: &Morpheme<Arc<PyDicData>>, py: Python<'py>) -> &'py PyString {
+        if self.matcher.matches_id(m.part_of_speech_id()) {
+            PyString::new(py, m.surface().deref())
+        } else {
+            PyString::new(py, m.normalized_form())
+        }
+    }
+}
+
+struct NormalizedNouns {
+    matcher: PosMatcher,
+}
+
+impl NormalizedNouns {
+    fn new<D: DictionaryAccess>(dic: &D) -> Self {
+        let matcher = make_matcher(dic, |p| p[5] == "*");
+        Self { matcher }
+    }
+}
+
+impl MorphemeProjection for NormalizedNouns {
+    fn project<'py>(&self, m: &Morpheme<Arc<PyDicData>>, py: Python<'py>) -> &'py PyString {
+        if self.matcher.matches_id(m.part_of_speech_id()) {
+            PyString::new(py, m.normalized_form())
+        } else {
+            PyString::new(py, m.surface().deref())
+        }
+    }
+}
+
+fn conjugating_matcher<D: DictionaryAccess>(dic: &D) -> PosMatcher {
+    make_matcher(dic, |pos| match pos[0].deref() {
+        "動詞" | "形容詞" | "助動詞" => true,
+        _ => false,
+    })
+}
+
+pub(crate) fn morpheme_projection<D: DictionaryAccess>(
+    projection: SurfaceProjection,
+    dict: &D,
+) -> Arc<dyn MorphemeProjection + Send + Sync> {
+    match projection {
+        // implement for surface to make this function full
+        SurfaceProjection::Surface => Arc::new(Surface {}),
+        SurfaceProjection::Normalized => Arc::new(Mapped {
+            func: |m| m.normalized_form(),
+        }),
+        SurfaceProjection::Reading => Arc::new(Mapped {
+            func: |m| m.reading_form(),
+        }),
+        SurfaceProjection::Dictionary => Arc::new(Mapped {
+            func: |m| m.dictionary_form(),
+        }),
+        SurfaceProjection::DictionaryAndSurface => Arc::new(DictionaryAndSurface::new(dict)),
+        SurfaceProjection::NormalizedAndSurface => Arc::new(NormalizedAndSurface::new(dict)),
+        SurfaceProjection::NormalizedNouns => Arc::new(NormalizedNouns::new(dict)),
+    }
+}
+
+fn make_matcher<D: DictionaryAccess, F: FnMut(&Vec<String>) -> bool>(
+    dic: &D,
+    mut f: F,
+) -> PosMatcher {
+    let ids = dic.grammar().pos_list.iter().enumerate().filter_map(|p| {
+        let (id, pos) = p;
+        if f(pos) {
+            Some(id as u16)
+        } else {
+            None
+        }
+    });
+    PosMatcher::new(ids)
+}
+
+pub(crate) fn resolve_projection(base: PyProjector, fallback: &PyProjector) -> PyProjector {
+    match (base, fallback) {
+        (None, None) => None,
+        (Some(p), _) => Some(p),
+        (_, Some(p)) => Some(p.clone()),
+    }
+}
+
+pub(crate) fn parse_projection<D: DictionaryAccess>(
+    value: &PyString,
+    dict: &D,
+) -> PyResult<PyProjector> {
+    value.to_str().and_then(|s| parse_projection_raw(s, dict))
+}
+
+pub(crate) fn parse_projection_raw<D: DictionaryAccess>(
+    value: &str,
+    dict: &D,
+) -> PyResult<PyProjector> {
+    match SurfaceProjection::try_from(value) {
+        Ok(v) => {
+            if v == SurfaceProjection::Surface {
+                Ok(None)
+            } else {
+                Ok(Some(morpheme_projection(v, dict)))
+            }
+        }
+        Err(e) => Err(crate::errors::SudachiError::new_err(format!(
+            "invalid surface projection: {e:?}"
+        ))),
+    }
+}
+
+pub(crate) fn parse_projection_opt<D: DictionaryAccess>(
+    value: Option<&PyString>,
+    dict: &D,
+) -> PyResult<PyProjector> {
+    match value {
+        None => Ok(None),
+        Some(v) => parse_projection(v, dict),
+    }
+}
diff --git a/python/src/tokenizer.rs b/python/src/tokenizer.rs
index 7912f1e4..7b84b9f2 100644
--- a/python/src/tokenizer.rs
+++ b/python/src/tokenizer.rs
@@ -25,7 +25,7 @@ use sudachi::dic::subset::InfoSubset;
 use sudachi::prelude::*;
 
 use crate::dictionary::PyDicData;
-use crate::morpheme::PyMorphemeListWrapper;
+use crate::morpheme::{PyMorphemeListWrapper, PyProjector};
 
 /// Unit to split text
 ///
@@ -68,14 +68,21 @@ impl From<PySplitMode> for Mode {
 
 /// Sudachi Tokenizer, Python version
 #[pyclass(module = "sudachipy.tokenizer", name = "Tokenizer")]
-pub struct PyTokenizer {
+pub(crate) struct PyTokenizer {
     tokenizer: StatefulTokenizer<Arc<PyDicData>>,
+    projection: PyProjector,
 }
 
 impl PyTokenizer {
-    pub(crate) fn new(dict: Arc<PyDicData>, mode: Mode, fields: InfoSubset) -> Self {
+    pub(crate) fn new(
+        dict: Arc<PyDicData>,
+        mode: Mode,
+        fields: InfoSubset,
+        projection: PyProjector,
+    ) -> Self {
         let mut tok = Self {
             tokenizer: StatefulTokenizer::new(dict, mode),
+            projection,
         };
         tok.tokenizer.set_subset(fields);
         tok
@@ -127,8 +134,10 @@ impl PyTokenizer {
 
         let out_list = match out {
             None => {
-                let morphemes = MorphemeList::empty(self.tokenizer.dict_clone());
-                let wrapper = PyMorphemeListWrapper::from(morphemes);
+                let dict = self.tokenizer.dict_clone();
+                let morphemes = MorphemeList::empty(dict);
+                let wrapper =
+                    PyMorphemeListWrapper::from_components(morphemes, self.projection.clone());
                 PyCell::new(py, wrapper)?
             }
             Some(list) => list,
diff --git a/sudachi/src/config.rs b/sudachi/src/config.rs
index 33804bff..e5e50a14 100644
--- a/sudachi/src/config.rs
+++ b/sudachi/src/config.rs
@@ -14,11 +14,14 @@
  * limitations under the License.
  */
 
+use std::convert::TryFrom;
 use std::env::current_exe;
 use std::fs::File;
 use std::io::BufReader;
 use std::path::{Path, PathBuf};
 
+use crate::dic::subset::InfoSubset;
+use crate::error::SudachiError;
 use lazy_static::lazy_static;
 use serde::Deserialize;
 use serde_json::Value;
@@ -96,6 +99,56 @@ impl PathResolver {
     }
 }
 
+#[derive(Deserialize, Clone, Copy, Debug, Eq, PartialEq)]
+#[serde(rename_all = "snake_case")]
+pub enum SurfaceProjection {
+    Surface,
+    Normalized,
+    Reading,
+    Dictionary,
+    DictionaryAndSurface,
+    NormalizedAndSurface,
+    NormalizedNouns,
+}
+
+impl Default for SurfaceProjection {
+    fn default() -> Self {
+        SurfaceProjection::Surface
+    }
+}
+
+impl SurfaceProjection {
+    /// Return required InfoSubset for the current projection type
+    pub fn required_subset(&self) -> InfoSubset {
+        match *self {
+            SurfaceProjection::Surface => InfoSubset::empty(),
+            SurfaceProjection::Normalized => InfoSubset::NORMALIZED_FORM,
+            SurfaceProjection::Reading => InfoSubset::READING_FORM,
+            SurfaceProjection::Dictionary => InfoSubset::DIC_FORM_WORD_ID,
+            SurfaceProjection::DictionaryAndSurface => InfoSubset::DIC_FORM_WORD_ID,
+            SurfaceProjection::NormalizedAndSurface => InfoSubset::NORMALIZED_FORM,
+            SurfaceProjection::NormalizedNouns => InfoSubset::NORMALIZED_FORM,
+        }
+    }
+}
+
+impl TryFrom<&str> for SurfaceProjection {
+    type Error = SudachiError;
+
+    fn try_from(value: &str) -> Result<Self, Self::Error> {
+        match value {
+            "surface" => Ok(SurfaceProjection::Surface),
+            "normalized" => Ok(SurfaceProjection::Normalized),
+            "reading" => Ok(SurfaceProjection::Reading),
+            "dictionary" => Ok(SurfaceProjection::Dictionary),
+            "dictionary_and_surface" => Ok(SurfaceProjection::DictionaryAndSurface),
+            "normalized_and_surface" => Ok(SurfaceProjection::NormalizedAndSurface),
+            "normalized_nouns" => Ok(SurfaceProjection::NormalizedNouns),
+            _ => Err(ConfigError::InvalidFormat(format!("unknown projection: {value}")).into()),
+        }
+    }
+}
+
 /// Setting data loaded from config file
 #[derive(Debug, Default, Clone)]
 pub struct Config {
@@ -109,15 +162,17 @@ pub struct Config {
     pub input_text_plugins: Vec<Value>,
     pub oov_provider_plugins: Vec<Value>,
     pub path_rewrite_plugins: Vec<Value>,
+    // this option is Python-only and is ignored in Rust APIs
+    pub projection: SurfaceProjection,
 }
 
 /// Struct corresponds with raw config json file.
 /// You must use filed names defined here as json object key.
 /// For plugins, refer to each plugin.
 #[allow(non_snake_case)]
-#[derive(Deserialize, Debug)]
+#[derive(Deserialize, Debug, Clone)]
 pub struct ConfigBuilder {
-    /// Analogue to Java Implementation path Override
+    /// Analogue to Java Implementation path Override    
     path: Option<PathBuf>,
     /// User-passed resourcePath
     #[serde(skip)]
@@ -126,14 +181,16 @@ pub struct ConfigBuilder {
     /// Is also automatically set on from_file
     #[serde(skip)]
     rootDirectory: Option<PathBuf>,
+    #[serde(alias = "system")]
     systemDict: Option<PathBuf>,
+    #[serde(alias = "user")]
     userDict: Option<Vec<PathBuf>>,
     characterDefinitionFile: Option<PathBuf>,
-
     connectionCostPlugin: Option<Vec<Value>>,
     inputTextPlugin: Option<Vec<Value>>,
     oovProviderPlugin: Option<Vec<Value>>,
     pathRewritePlugin: Option<Vec<Value>>,
+    projection: Option<SurfaceProjection>,
 }
 
 pub fn default_resource_dir() -> PathBuf {
@@ -151,6 +208,12 @@ pub fn default_config_location() -> PathBuf {
     resdir
 }
 
+macro_rules! merge_cfg_value {
+    ($base: ident, $o: ident, $name: tt) => {
+        $base.$name = $base.$name.or_else(|| $o.$name.clone())
+    };
+}
+
 impl ConfigBuilder {
     pub fn from_opt_file(config_file: Option<&Path>) -> Result<Self, ConfigError> {
         match config_file {
@@ -236,8 +299,24 @@ impl ConfigBuilder {
             input_text_plugins: self.inputTextPlugin.unwrap_or(Vec::new()),
             oov_provider_plugins: self.oovProviderPlugin.unwrap_or(Vec::new()),
             path_rewrite_plugins: self.pathRewritePlugin.unwrap_or(Vec::new()),
+            projection: self.projection.unwrap_or(SurfaceProjection::Surface),
         }
     }
+
+    pub fn fallback(mut self, other: &ConfigBuilder) -> ConfigBuilder {
+        merge_cfg_value!(self, other, path);
+        merge_cfg_value!(self, other, resourcePath);
+        merge_cfg_value!(self, other, rootDirectory);
+        merge_cfg_value!(self, other, systemDict);
+        merge_cfg_value!(self, other, userDict);
+        merge_cfg_value!(self, other, characterDefinitionFile);
+        merge_cfg_value!(self, other, connectionCostPlugin);
+        merge_cfg_value!(self, other, inputTextPlugin);
+        merge_cfg_value!(self, other, oovProviderPlugin);
+        merge_cfg_value!(self, other, pathRewritePlugin);
+        merge_cfg_value!(self, other, projection);
+        self
+    }
 }
 
 impl Config {
@@ -398,4 +477,21 @@ mod tests {
         assert!(npath[0].starts_with(path_dir));
         Ok(())
     }
+
+    #[test]
+    fn config_builder_fallback() {
+        let mut cfg = ConfigBuilder::empty();
+        cfg.path = Some("test".into());
+        let cfg2 = ConfigBuilder::empty();
+        let cfg2 = cfg2.fallback(&cfg);
+        assert_eq!(cfg2.path, Some("test".into()));
+    }
+
+    #[test]
+    fn surface_projection_tryfrom() {
+        assert_eq!(
+            SurfaceProjection::Surface,
+            SurfaceProjection::try_from("surface").unwrap()
+        );
+    }
 }
diff --git a/sudachi/src/pos.rs b/sudachi/src/pos.rs
index 7862a75b..971428eb 100644
--- a/sudachi/src/pos.rs
+++ b/sudachi/src/pos.rs
@@ -17,6 +17,7 @@
 use crate::util::fxhash::FxBuildHasher;
 use std::collections::HashSet;
 
+#[derive(Clone)]
 pub struct PosMatcher {
     ids: HashSet<u16, FxBuildHasher>,
 }


Test patch:
diff --git a/python/tests/resources/test_config_template.json b/python/tests/resources/test_config_template.json
deleted file mode 100644
index 4e1a6273..00000000
--- a/python/tests/resources/test_config_template.json
+++ /dev/null
@@ -1,24 +0,0 @@
-{
-  "path" : "tests/resources/",
-  "systemDict" : "$system_dict",
-  "userDict" : "$user_dict",
-  "characterDefinitionFile" : "char.def",
-  "inputTextPlugin" : [
-    { "class" : "com.worksap.nlp.sudachi.DefaultInputTextPlugin" }
-  ],
-  "oovProviderPlugin" : [
-    { "class" : "com.worksap.nlp.sudachi.SimpleOovPlugin",
-      "oovPOS" : [ "名詞", "普通名詞", "一般", "*", "*", "*" ],
-      "leftId" : 8,
-      "rightId" : 8,
-      "cost" : 6000 }
-  ],
-  "pathRewritePlugin" : [
-    { "class" : "com.worksap.nlp.sudachi.JoinNumericPlugin",
-      "enableNormalize" : true },
-    { "class" : "com.worksap.nlp.sudachi.JoinKatakanaOovPlugin",
-      "oovPOS" : [ "名詞", "普通名詞", "一般", "*", "*", "*" ],
-      "minLength" : 3
-    }
-  ]
-}
diff --git a/python/tests/test_build.py b/python/tests/test_build.py
index 7d4553e1..4051b47f 100644
--- a/python/tests/test_build.py
+++ b/python/tests/test_build.py
@@ -17,10 +17,20 @@
 from pathlib import Path
 
 import sudachipy
+from sudachipy.config import Config
+from dataclasses import replace
 
 FILE_PATH = Path(__file__)
 RESOURCES_PATH = FILE_PATH.parent / "resources"
-CONFIG_TEMPLATE = RESOURCES_PATH / "test_config_template.json"
+CFG_TEMPLATE = Config(
+    oovProviderPlugin=[
+        { "class" : "com.worksap.nlp.sudachi.SimpleOovPlugin",
+          "oovPOS" : [ "名詞", "普通名詞", "一般", "*", "*", "*" ],
+          "leftId" : 8,
+          "rightId" : 8,
+          "cost" : 6000 }
+    ]
+)
 
 
 class MyTestCase(unittest.TestCase):
@@ -37,19 +47,6 @@ def tearDown(self) -> None:
         Path(self.tmpdir).rmdir()
         super().tearDown()
 
-    def make_config(self, system, user):
-        with open(CONFIG_TEMPLATE, encoding="utf-8") as fin:
-            template_data = fin.read()
-        system = system.replace("\\", "/")
-        user = ['"' + str(Path(u).name).replace("\\", "/") + '"' for u in user]
-        template_data = template_data.replace("$system_dict", str(Path(system).name))
-        template_data = template_data.replace("\"$user_dict\"", "[" + ", ".join(user) + "]")
-        tempfname = tempfile.mktemp(prefix="sudachi_cfg", suffix=".json", dir=self.tmpdir)
-        with open(tempfname, "wt", encoding='utf-8') as f:
-            f.write(template_data)
-        self.tempfiles.append(tempfname)
-        return tempfname
-
     def test_build_system(self):
         out_tmp = tempfile.mktemp(prefix="sudachi_sy", suffix=".dic", dir=self.tmpdir)
         self.tempfiles.append(out_tmp)
@@ -59,7 +56,7 @@ def test_build_system(self):
             output=out_tmp
         )
         self.assertIsNotNone(stats)
-        cfg = self.make_config(out_tmp, [])
+        cfg = replace(CFG_TEMPLATE, system=out_tmp)
         dict = sudachipy.Dictionary(config_path=cfg)
         tok = dict.create()
         result = tok.tokenize("東京にいく")
@@ -80,8 +77,9 @@ def test_build_user1(self):
             lex=[RESOURCES_PATH / "user1.csv"],
             output=u1_dic
         )
-        cfg = self.make_config(sys_dic, [u1_dic])
-        dict = sudachipy.Dictionary(config_path=cfg)
+
+        cfg = replace(CFG_TEMPLATE, system=sys_dic, user=[u1_dic])
+        dict = sudachipy.Dictionary(config=cfg)
         tok = dict.create()
         result = tok.tokenize("すだちにいく")
         self.assertEqual(result.size(), 3)
@@ -111,7 +109,7 @@ def test_build_user2(self):
             output=u2_dic
         )
 
-        cfg = self.make_config(sys_dic, [u1_dic, u2_dic])
+        cfg = replace(CFG_TEMPLATE, system=sys_dic, user=[u1_dic, u2_dic])
         dict = sudachipy.Dictionary(config_path=cfg)
         tok = dict.create()
         result = tok.tokenize("かぼすにいく")
diff --git a/python/tests/test_pretokenizers.py b/python/tests/test_pretokenizers.py
index 37e0dbbc..1c31b449 100644
--- a/python/tests/test_pretokenizers.py
+++ b/python/tests/test_pretokenizers.py
@@ -19,6 +19,7 @@
 
 import sudachipy
 from sudachipy import MorphemeList
+from sudachipy.config import Config
 
 
 class PretokenizerTestCase(unittest.TestCase):
@@ -72,5 +73,36 @@ def _handler(index, sentence: tokenizers.NormalizedString, ml: MorphemeList):
         self.assertEqual(res.ids, [6, 7])
 
 
+    def test_with_projection(self):
+        pretok = self.dict.pre_tokenizer(sudachipy.SplitMode.A, projection="reading")
+        vocab = {
+            "[UNK]": 0,
+            "ノム": 1,
+            "サケ": 2,
+            "ヒト": 3,
+            "ヲ": 5,
+        }
+        tok = tokenizers.Tokenizer(WordLevel(vocab, unk_token="[UNK]"))
+        tok.pre_tokenizer = pretok
+        res = tok.encode("酒を飲む人")
+        self.assertEqual([2, 5, 1, 3], res.ids)
+
+    def test_projection_surface_override(self):
+        dictobj = sudachipy.Dictionary(config=sudachipy.config.Config(projection="reading"))
+        pretok = dictobj.pre_tokenizer(sudachipy.SplitMode.A, projection="surface")
+        vocab = {
+            "[UNK]": 0,
+            "酒": 1,
+            "人": 2,
+            "飲む": 3,
+            "を": 5,
+            "外国人参政権": 4
+        }
+        tok = tokenizers.Tokenizer(WordLevel(vocab, unk_token="[UNK]"))
+        tok.pre_tokenizer = pretok
+        res = tok.encode("酒を飲む人")
+        self.assertEqual(res.ids, [1, 5, 3, 2])
+
+
 if __name__ == '__main__':
     unittest.main()
diff --git a/python/tests/test_projection.py b/python/tests/test_projection.py
new file mode 100644
index 00000000..c078bf06
--- /dev/null
+++ b/python/tests/test_projection.py
@@ -0,0 +1,93 @@
+#   Copyright (c) 2023 Works Applications Co., Ltd.
+#
+#   Licensed under the Apache License, Version 2.0 (the "License");
+#   you may not use this file except in compliance with the License.
+#   You may obtain a copy of the License at
+#
+#       http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#   distributed under the License is distributed on an "AS IS" BASIS,
+#   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#   See the License for the specific language governing permissions and
+#   limitations under the License.
+
+import unittest
+
+from sudachipy import Dictionary
+from sudachipy.config import Config
+
+class MyTestCase(unittest.TestCase):
+    def setUp(self) -> None:
+        self.dict = Dictionary(config=Config(projection="reading"))
+
+    def test_projection_surface_override(self):
+        tok = self.dict.create(projection="surface")
+        morphs = tok.tokenize("酒を飲む人")
+        self.assertEqual(4, morphs.size())
+        self.assertEqual("酒", morphs[0].surface())
+        self.assertEqual("を", morphs[1].surface())
+        self.assertEqual("飲む", morphs[2].surface())
+        self.assertEqual("人", morphs[3].surface())
+
+    def test_projection_reading(self):
+        tok = self.dict.create()
+        morphs = tok.tokenize("酒を飲む人")
+        self.assertEqual(4, morphs.size())
+        self.assertEqual("サケ", morphs[0].surface())
+        self.assertEqual("ヲ", morphs[1].surface())
+        self.assertEqual("ノム", morphs[2].surface())
+        self.assertEqual("ヒト", morphs[3].surface())
+
+    def test_projection_dictionary(self):
+        tok = self.dict.create(projection="dictionary")
+        morphs = tok.tokenize("酒を飲まなかった人")
+        self.assertEqual(6, morphs.size())
+        self.assertEqual("酒", morphs[0].surface())
+        self.assertEqual("を", morphs[1].surface())
+        self.assertEqual("飲む", morphs[2].surface())
+        self.assertEqual("ない", morphs[3].surface())
+        self.assertEqual("た", morphs[4].surface())
+        self.assertEqual("人", morphs[5].surface())
+
+    def test_projection_normalized(self):
+        tok = self.dict.create(projection="normalized")
+        morphs = tok.tokenize("MEGAへ行く")
+        self.assertEqual(3, morphs.size())
+        self.assertEqual("メガ", morphs[0].surface())
+        self.assertEqual("MEGA", morphs[0].raw_surface())
+        self.assertEqual("へ", morphs[1].surface())
+        self.assertEqual("行く", morphs[2].surface())
+
+    def test_projection_dictionary_and_surface(self):
+        tok = self.dict.create(projection="dictionary_and_surface")
+        morphs = tok.tokenize("酒を飲まなかった人")
+        self.assertEqual(6, morphs.size())
+        self.assertEqual("酒", morphs[0].surface())
+        self.assertEqual("を", morphs[1].surface())
+        self.assertEqual("飲ま", morphs[2].surface())
+        self.assertEqual("なかっ", morphs[3].surface())
+        self.assertEqual("た", morphs[4].surface())
+        self.assertEqual("人", morphs[5].surface())
+
+    def test_projection_normalized_and_surface(self):
+        tok = self.dict.create(projection="normalized_and_surface")
+        morphs = tok.tokenize("MEGAへ行こう")
+        self.assertEqual(3, morphs.size())
+        self.assertEqual("メガ", morphs[0].surface())
+        self.assertEqual("MEGA", morphs[0].raw_surface())
+        self.assertEqual("へ", morphs[1].surface())
+        self.assertEqual("行こう", morphs[2].surface())
+
+    def test_projection_normalized_nouns(self):
+        tok = self.dict.create(projection="normalized_nouns")
+        morphs = tok.tokenize("MEGAへ行こう")
+        self.assertEqual(3, morphs.size())
+        self.assertEqual("メガ", morphs[0].surface())
+        self.assertEqual("MEGA", morphs[0].raw_surface())
+        self.assertEqual("へ", morphs[1].surface())
+        self.assertEqual("行こう", morphs[2].surface())
+
+
+if __name__ == '__main__':
+    unittest.main()