Package: unidecode-replace
Unidecode ReplaceThis is a Python package providing search-and-replace in a unidecode compatible manner. This means that, when searching,unidecodeequivalences are taken into account, but the replacing is still done on the original string.For example, German character "ö" unidecodes as "o". So, when searching a text in German language, one could search for "o" and expect to find every "o", but also every "ö" (the opposite is also true: searching for "ö" would find both of these characters). Unidecode Replace will do this:>>>fromunidecode_replaceimportunidecode_replace>>>unidecode_replace("Übergröße","ö","!!!")'Übergr!!!ße'>>>unidecode_replace("Übergröße","o","!!!")'Übergr!!!ße'>>>unidecode_replace("Just an ordinary 'o'","ö","!!!")"Just an !!!rdinary '!!!'"Note:If you like the features described in this document, but you want a regular (not unidecode-compatible) search, there is a flag for that.ContentMain functionsunidecode_replaceunidecode_wrapMain argumentsSearch(es)Substitution(s)Auxiliary functions fromunidecodeExposed internalsMain functionsunidecode_replaceThe core of the package isunidecode_replacefunction, which is a wrapper for the classUnidecodeReplacethat does all the work. It's signature is this:defunidecode_replace(string:str,search:SearchT,sub:SubT,*,allow_overlaps:bool=False,re_search:bool=False,re_flags:re.RegexFlag=re.RegexFlag(0),count:Optional[int]=None,pos:Optional[int]=None,endpos:Optional[int]=None,unidecoded_search:bool=True,str_case_sensitive:bool=True,)->str:...The arguments are as follows:string: The input string on which the search and replace are performed.search: A search string or a regex, or a sequence of them. For more details, seeSearch(es).sub: A replacement string or a function, or a sequence of them. For more details, seeSubstitution(s).allow_overlaps: IfTrue, overlapping instances of found text are replaced.re_search: IfTrueandsearchis a string, it'll be compiled as a regular expression. If it's a sequence, any string in it will also be compiled.re_flags: Flags used when compiling regular expressions (ifre_searchis set toTrue).count: If set, only the firstcountoccurrences are replaced.pos,endpos: If defined, search and replace will only be done from positionpos(default: 0) toendpos - 1(defaulting to the end ofstring).unidecoded_search: If set toFalse, the search will be literal (i.e., not based on the unidecoded version ofstring). This behaves equivalently tostr.replaceandre.sub, but with the added benefit of doing multiple searches in one pass and allowing one to useposandendpos.str_case_sensitive: If set toTrue, string searches will be performed in case-sensitive manner. If set toFalse, string searches will be performed in case-insensitive manner. This setting does not affect regex searching.The function returns a copy ofstringwith matched substrings replaced.unidecode_wrapThis function does pretty much the same thing asunidecode_replace, except that itssubis replaced by two arguments:prefixandsuffix:defunidecode_wrap(string:str,search:SearchT,prefix:str,suffix:str,*,allow_overlaps:bool=False,re_search:bool=False,re_flags:re.RegexFlag=re.RegexFlag(0),count:Optional[int]=None,pos:Optional[int]=None,endpos:Optional[int]=None,unidecoded_search:bool=True,str_case_sensitive:bool=True,)->str:...What it does is wrapping each found substring with two values. The main purpose is easy marking of what was found. For example,>>>unidecode_wrap("This is some string.","some","<b>","</b>")'This is <b>some</b> string.'Main argumentsLet us consider two main arguments - search(es) and substitution(s) - in more detail.Search(es)As mentioned above, the search argumentstringcan be a string, a regular expression (re.Pattern), or a sequence of one or more of these.If it is a string, then:Ifre_searchis set toFalse, it will be matched tostringas it is.Ifre_searchis set toTrue, it will be compiled as a regular expression before being matched tostring.If it is a regular expression orre_searchis set toTrue, the matching will be the usual regex one.If it is a regex pattern (i.e., a compiled regex), the matching is done as one would expect of regex searches.If it is a sequence, each item (a string or a regex pattern) is treated as just described.Ifstr_case_sensitiveflag is set toFalse, all string matches are done in case-insensitive manner, and if it'sTruethen they are case-sensitive. This setting does not affect regular expressions.When a string is compiled as a regex (becausere_searchis set toTrue), the value ofre_flagsis used asflagsinre.compile. For example, includere.Ihere to get case-insensitive search. However, for regex pattern searches (i.e., those that are already compiled),re_searchandre_flagshave no effect.To avoid unidecode-compatible searching (for example, treating "ö" and "o" as the same character), setunidecoded_searchtoFalse.Substitution(s)As mentioned above, the substitution argumentsubcan be a string or a callable or a sequence of these things. Let us see how these act one-on-one (i.e., when there is only onesearchand onesubitem).Ifsearchis a string andsubis a string, than any matched substring ofstringwill be replaced bysub.Ifsearchis a string andsubis a callable, it should accept one string (which will have the found substring provided) and any matched substring ofstringwill be replaced by whatever that callable returns.Ifsearchis a regex pattern andsubis a string, than any matched substring ofstringwill be replaced by the expandedsub(i.e., any regex groups in it will be replaced by the values of those groups).Ifsearchis a regex pattern andsubis a callable, it should accept one argumentmand any matched substring ofstringwill be replaced by whatever that callable returns.The argumentmmentioned in item #4 is aUnidecodeReMatchinstance. This can be used as a normalre.Matchas it wraps the original one (accessible asm.original), but it will be properly mapped tostring(m.originalhas its values matching the unidecoded version ofstring).If given as sequences,searchandsubshould be of the same length and they will match 1-to-1, meaning that the first search item will be replaced with the first substitution item, the second search item will be replaced with the second substitution item, etc.There is one exception to this rule: ifsearchis a sequence, thensubcan still be a single item or a sequence containing only one item. In this special case, all matched search items will be substituted by that same one substitution item. This is effectively used byunidecode_wrap, which internally provides a single callablesub.Auxiliary functions fromunidecodeThere are two auxiliary functions whose functionality is more related tounidecode, but they are exposed in this package:can_be_unidecoded(string: str) -> bool: ReturnTrueif thestringcan be unidecoded without errors.get_invalid_chars(string: str) -> set[str]: Return a set of chars instringthat fail to unidecode.Exposed internalsApart fromunidecode_replaceandunidecode_wrap, the package also exposes the following classes (but most of them won't be imported withfrom unidecode_replace import *):UnidecodeReplace: The class that does all the actual work. Sinceunidecode_replaceexposes all of its functionality, you'd only want to use this if you were to inherit it.SearchItem,SearchItemStr,SearchItemRegex: Internal classes that hold(search, sub)pairs and perform the actual searching and generating the replacement strings. You probably don't need these for more than just type annotations in the event of inheritingUnidecodeReplaceand extending its search capabilities.UnidecodeReMatch: The wrapper class forre.Match, used to map the results of regular expressions performed on unidecoded strings back to the original ones. Also unlikely to be used for more than type annotations, but this one is exposed in__all__. The reason for this is that one might need it for type annotations in the callables used as substituted when searching with regular expressions.SearchT,SubT,u2iT: Types forsearchandsubarguments, and foru2iattribute (an internal mapping used byUnidecodeReplaceto map indexes from unidecoded string to the original (input) one).