Package: budou
English text has many clues, like spacing and hyphenation, that enable beautiful
and legible line breaks. Some CJK languages lack these clues, and so are
notoriously more difficult to process. Without a more careful approach,
breaks can occur randomly and usually in the middle of a word. This is a
long-standing issue with typography on the web and results in a degradation
of readability.Budou automatically translates CJK sentences into HTML with
lexical chunks wrapped in non-breaking markup, so as to semantically control line
breaks. Budou uses word segmenters to analyze input sentences. It can also
concatenate proper nouns to produce meaningful chunks utilizing
part-of-speech (pos) tagging and other syntactic information. Processed chunks are
wrapped with theSPANtag. These semantic units will no longer be split at
the end of a line if given a CSSdisplayproperty set toinline-block.InstallationThe package is listed in the Python Package Index (PyPI), so you can install it
with pip:$pipinstallbudouOutputBudou outputs an HTML snippet wrapping chunks withspantags:<span><spanclass="ww">常に</span><spanclass="ww">最新、</span><spanclass="ww">最高の</span><spanclass="ww">モバイル。</span></span>Semantic chunks in the output HTML will not be split at the end of line by
configuring eachspantag withdisplay: inline-blockin CSS..ww{display:inline-block;}By using the output HTML from Budou and the CSS above, sentences
on your webpage will be rendered with legible line breaks:Using as a command-line appYou can process your text by running thebudoucommand:$budou渋谷のカレーを食べに行く。The output is:<span><spanclass="ww">渋谷の</span><spanclass="ww">カレーを</span><spanclass="ww">食べに</span><spanclass="ww">行く。</span></span>You can also configure the command with optional parameters.
For example, you can change the backend segmenter toMeCaband change the
class name towordwrapby running:$budou渋谷のカレーを食べに行く。--segmenter=mecab--classname=wordwrapThe output is:<span><spanclass="wordwrap">渋谷の</span><spanclass="wordwrap">カレーを</span><spanclass="wordwrap">食べに</span><spanclass="wordwrap">行く。</span></span>Run the help commandbudou -hto see other available options.Using programmaticallyYou can use thebudou.parsemethod in your Python scripts.importbudouresults=budou.parse('渋谷のカレーを食べに行く。')print(results['html_code'])# <span><span class="ww">渋谷の</span><span class="ww">カレーを</span># <span class="ww">食べに</span><span class="ww">行く。</span></span>You can also make a parser instance to reuse the segmenter backend with the same
configuration. If you want to integrate Budou into your web development
framework in the form of a custom filter or build process, this would be the way
to go.importbudouparser=budou.get_parser('mecab')results=parser.parse('渋谷のカレーを食べに行く。')print(results['html_code'])# <span><span class="ww">渋谷の</span><span class="ww">カレーを</span># <span class="ww">食べに</span><span class="ww">行く。</span></span>forchunkinresults['chunks']:print(chunk.word)# 渋谷の 名詞# カレーを 名詞# 食べに 動詞# 行く。 動詞(deprecated)authenticatemethodauthenticate, which had been the method used to create a parser in
previous releases, is now deprecated.
Theauthenticatemethod is now a wrapper around theget_parsermethod
that returns a parser with theGoogle Cloud Natural Language APIsegmenter backend.
The method is still available, but it may be removed in a future release.importbudouparser=budou.authenticate('/path/to/credentials.json')# This is equivalent to:parser=budou.get_parser('nlapi',credentials_path='/path/to/credentials.json')Available segmenter backendsYou can choose different segmenter backends depending on the needs of
your environment. Currently, the segmenters below are supported.NameIdentifierSupported LanguagesGoogle Cloud Natural Language APInlapiChinese, Japanese, KoreanMeCabmecabJapaneseTinySegmentertinysegmenterJapaneseSpecify the segmenter when you run thebudoucommand or load a parser.
For example, you can run thebudoucommand with the MeCab segmenter by
passing the--segmenter=mecabparameter:$budou今日も元気です--segmenter=mecabYou can passsegmenterparameter when you load a parser:importbudouparser=budou.get_parser('mecab')parser.parse('今日も元気です')If no segmenter is specified, the Google Cloud Natural Language API is used as
the default.Google Cloud Natural Language API SegmenterThe Google Cloud Natural Language API (https://cloud.google.com/natural-language/)
(NL API) analyzes input sentences using
machine learning technology. The API can extract not only syntax but also
entities included in the sentence, which can be used for better quality
segmentation (see more atEntity mode). Since this is a simple
REST API, you don’t need to maintain a dictionary. You can also support multiple
languages using one single source.Supported languagesSimplified Chinese (zh)Traditional Chinese (zh-Hant)Japanese (ja)Korean (ko)For those considering using Budou for Korean sentences, please refer to
theKorean supportsection.AuthenticationThe NL API requires authentication before use. First, create a Google Cloud Platform
project and enable the Cloud Natural Language API. Billing also needs to be enabled
for the project. Then, download a credentials file for a service account by
accessing theGoogle Cloud Consoleand navigating through “API & Services” > “Credentials” > “Create credentials” >
“Service account key” > “JSON”.Budou will handle authentication once the path to the credentials file is set
in theGOOGLE_APPLICATION_CREDENTIALSenvironment variable.$exportGOOGLE_APPLICATION_CREDENTIALS='/path/to/credentials.json'You can also pass the path to the credentials file when you initialize the
parser.parser=budou.get_parser('nlapi',credentials_path='/path/to/credentials.json')The NL API segmenter usesSyntax Analysisand incurs costs according to
monthly usage. The NL API has free quota to start testing the feature without charge.
Please refer tohttps://cloud.google.com/natural-language/pricingfor more
detailed pricing information.Caching systemParsers using the NL API segmenter cache responses from the API in order to
prevent unnecessary requests to the API and to make processing faster. If you want to
force-refresh the cache, setuse_cachetoFalse.parser=budou.get_parser(segmenter='nlapi',use_cache=False)result=parser.parse('明日は晴れるかな')In theGoogle App Engine Python 2.7 Standard Environment,
Budou tries to use thememcacheservice to cache output efficiently across instances.
In other environments, Budou creates a cache file in thepython pickleformat in
your file system.Entity modeThe default parser only uses results from Syntactic Analysis for parsing, but you
can also utilize results fromEntity Analysisby specifyinguse_entity=True.
Entity Analysis will improve the accuracy of parsing for some phrases,
especially proper nouns, so it is recommended if your target sentences
include names of individual people, places, organizations, and so on.Please note that Entity Analysis will result in additional pricing because it
requires additional requests to the NL API. For more details about API pricing,
please refer tohttps://cloud.google.com/natural-language/pricing.importbudou# Without Entity mode (default)result=budou.parse('六本木ヒルズでご飯を食べます。',use_entity=False)print(result['html_code'])# <span class="ww">六本木</span><span class="ww">ヒルズで</span># <span class="ww">ご飯を</span><span class="ww">食べます。</span># With Entity moderesult=budou.parse('六本木ヒルズでご飯を食べます。',use_entity=True)print(result['html_code'])# <span class="ww">六本木ヒルズで</span># <span class="ww">ご飯を</span><span class="ww">食べます。</span>MeCab SegmenterMeCab (https://github.com/taku910/mecab) is an open source text segmentation
library for the Japanese language. Unlike the Google Cloud Natural Language API segmenter,
the MeCab segmenter does not require any billed API calls, so you can process
sentences for free and without an internet connection. You can also customize the
dictionary by building your own.Supported languagesJapaneseInstallationYou need to have MeCab installed to use the MeCab segmenter in Budou.
You can install MeCab with an IPA dictionary by running$makeinstall-mecabin the project’s home directory after cloning this repository.TinySegmenter-based SegmenterTinySegmenter (http://chasen.org/~taku/software/TinySegmenter/) is a compact
Japanese tokenizer originally created by (c) 2008 Taku Kudo.
It tokenizes sentences by matching against a combination of patterns carefully
designed using machine learning. This means thatyou can use this backend
without any additional setup!Supported languagesJapaneseKorean supportKorean has spaces between chunks, so you can perform line breaking simply by
puttingword-break: keep-allin your CSS. We recommend that you use this
technique instead of using Budou.Use casesBudou is designed to be used mostly in eye-catching sentences such as titles
and headings on the assumption that split chunks would stand out negatively
at larger font sizes.AccessibilitySome screen reader software packages read Budou’s wrapped chunks one by one.
This may degrade the user experience for those who need audio support.
You can attach any attribute to the output chunks to enhance accessibility.
For example, you can make screen readers read undivided sentences by
combining thearia-describedbyandaria-labelattributes in the output.<pid="description"aria-label="やりたいことのそばにいる"><spanclass="ww"aria-describedby="description">やりたい</span><spanclass="ww"aria-describedby="description">ことの</span><spanclass="ww"aria-describedby="description">そばに</span><spanclass="ww"aria-describedby="description">いる</span></p>This functionality is currently nonfunctionaldue to the html5lib sanitizer’s
behavior, which strips ARIA-related attributes from the output HTML. Progress on this
issue is tracked athttps://github.com/google/budou/issues/74AuthorShuhei IitsukaWebsite:https://tushuhei.comTwitter:https://twitter.com/tushuheiDisclaimerThis library is authored by a Googler and copyrighted by Google, but is not an
official Google product.LicenseCopyright 2018 Google LLCLicensed under the Apache License, Version 2.0 (the “License”);
you may not use this file except in compliance with the License.
You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an “AS IS” BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.