/
6&:*1

1RWHRQVRIWK\SKHQVLQ8QLFRGH
'RFXPHQW7\SH:RUNLQJ*URXS'RFXPHQW
6RXUFH.HQW.DUOVVRQ6WDWXV([SHUW&RQWULEXWLRQ'DWH

 &KDUWJO\SKVIRUVRIWK\SKHQV
The chart glyphs for DOOWKUHH soft hyphens allocated so far (SOFT HYPHEN (00AD),ARMENIAN HYPHEN (058A), and MONGOLIAN TODO SOFT HYPHEN (1806)) shouldhave a dashed box with suitable indication inside.
Side note: It is possible that one should allocate no-break hyphens for Armenian andMongolian, see below, but this document does not suggest that.

 $XWRPDWLFK\SKHQDWLRQYVPDQXDOLQWHUYHQWLRQZLWKDXWRPDWLFK\SKHQDWLRQ
Automatic hyphenators (as well as higher level protocols for indication hyphenationpoints) should have at least three levels of preferredness for hyphenation points: wordboundary (highest priority), morpheme boundary (within a word, middle priority), andsyllable boundary (within a morpheme, lowest priority). This may be indicated by theauthor in a separate table, giving such priorities in a higher-level protocol, or in adictionary with hyphenation indications for the assumed or declared language of the text,
as well as heuristics.
Unfortunately, using system provided dictionaries and heuristics may sometimes
lead to erroneous hyphenations, and the set of failures may vary between systems.
However, for interpreting the soft hyphens, which are most likely to be inserted
manually at the points of failure of the automatic hyphenator, there is an ABSOLUTE
need for determinism. There must be no dependence on dictionaries or heuristics in theinterpretation of soft hyphens, precisely because soft hyphens are most likely to occur at
the points where the dictionaries and heuristics failed. In the suggested new text on softhyphens given below, default deterministic rules are given for soft hyphens, and they arebased on the actual spell change rules that are used for composite words. Trying to"reverse" them would lead to non-determinism that cannot be reliably resolved into
determinism even by the best dictionaries or heuristics.
The rules for soft hyphens should be dealt with by rendering software libraries,
deleting the soft hyphen just before display (not in the source text) and doing spellchanges (for the display, not in the source text), both according to the default rules
below or some tailoring of them for a particular orthography. Fonts should have theappropriate hyphen glyph for the soft hyphen characters.

1

Automatic hyphenators may still be applied to the word parts adjoined by explicit softhyphens, but should then only generate hyphenation points of low or middle priority, nothigh priority.
In contrast, the US suggestion (in their comments to the second amendment to10646-1:2000) of letting soft hyphens always be invisible when not causing a line break,and then incur no spell changes, while all spell changes would be done when softhyphens cause a line break, has a number of serious problems, that are unsolvable in a
deterministic fashion without the use of a higher level protocol:a.

It would make it impossible, without the use of the private use area or new
character allocations, to display Armenian and Mongolian soft hyphens.

b.

It would make the insertion of extra letters non-deterministic even with the helpof the best dictionaries, since the insertion or not of an extra letter can change
the meaning (between two legitimate meanings) of a word.

c.

It would make the uppercasing of a letter after the soft hyphen not only
language dependent (German), but also non-deterministic even with the helpof the best dictionaries, since it may still not be able to determine correctly
whether (a prefix of) the rest of the word is a noun or not.

It is therefore LPSHUDWLYH that the rules for soft hyphens in plain text are deterministic,given a language/orthography. There should also be good GHIDXOW rules in case thelanguage/orthography is wrong or unknown. It would be to go too far to have thelanguage/orthography specific exceptions listed in 10646, but the default rules should begiven. For almost all texts, the default rules (given below) will be sufficient.
As shown in this paper (see the suggested text on soft hyphens below), all SHY
exceptions to “just invisible vs. just hyphen plus line break” that are known to the author
can be solved reliably without a higher level protocol about hyphenation, but possiblywith information about which orthography, though the latter information is rarely needed.
The interpretation of the SOFT HYPHEN in programs currently varies greatly. Some
programs always display it as a hyphen, some let it be invisible, but never let it cause aline break, even though the text is otherwise automatically line broken, some do allow
the line to get broken at a SHY then showing it while it is otherwise invisible, and some
programs completely misbehave, e.g. replacing the soft hyphen with a hard line break(no hyphen at all). Only with a good and clear as possible specification on how softhyphens should be interpreted is there any hope to get them to work properly in a wider
range of applications. While doing so, it is important to make soft hyphens work wellalso for Swedish, German and other languages where composite words are common,
not just for English, as well as being able to write texts about hyphenation in Armenianand Mongolian. The text below does not, however, specify how soft hyphens should beinterpreted if markup intervenes, e.g. LWDOLF!NUlSSLWDOLF!SDSSHU, where - here is a softhyphen. It is in such cases probably best to render it visibly.
The US proposed new text about soft hyphen also misrepresents the ARMENIAN
and MONGOLIAN soft hyphens as possible renderings of the SOFT HYPHEN, whilst
they are actually soft hyphens themselves. The SOFT HYPHEN should be rendered asa HYPHEN (and NO-BREAK HYPHEN) when visible. The glyphs for ARMENIAN
HYPHEN and MONGOLIAN TODO SOFT HYPHEN are different, while there are nocorresponding non-soft hyphen characters for them.
The suggested text below, on the other hand, is in line with and clarifies the current
text on soft hyphens in 10646.

2

 6XJJHVWHGQHZWH[WRQVRIWK\SKHQVIRUDQQH[)RI
6LGHUHPDUN7KHW\SRJUDSKLFVW\OHVXVHGLQDQQH[HV)DQG3VKRXOGEHPDGHWKHVDPH
SOFT HYPHEN (00AD) (SHY), ARMENIAN HYPHEN (058A) (ASHY), andMONGOLIAN TODO SOFT HYPHEN (1806) (MSHY): These characters are format
characters, as well as graphic characters, that indicate preferred intra-word hyphenationand line-break points in the text. SHY is used for the Latin, Greek, and Cyrillic scripts.
Note that an automatic hyphenator should use at least three levels of preferredness. An
explicit soft hyphen should be interpreted as a high priority hyphenation point (just as
HYPHEN should be), even if that is orthographically incorrect.
If the line LVQRWbroken at the point indicated by the soft hyphen character, the softhyphen is often not rendered (see the rules below). The soft hyphen may then also
suppress or change to lowercase the following character. The default rules for display
and spell changes for non-line-breaking occurrences of soft hyphens are as follows:Firstly try to match (rare):[[<SHY>[[[[[[[<SHY>;;;;;<SHY>;
Secondly try to match (common):[<SHY>][][][<SHY>=;];<SHY>];<SHY>=;=
DE
D<ASHY>E
PQ
P<MSHY>Q
Otherwise (less common): Keep the soft hyphen and display it with a visible glyph.
In these rules, ; and = are uppercase Latin/Greek/Cyrillic letters without a compatibility
decomposition in Unicode, [ and ] are the respective corresponding lowercase letters;for [ and ;, “letters” here also includes combining characters from the 03xx-block withthe case being derived from the preceding base letter (if any); D and E are Armenian
letters, and P and Q are Mongolian letters (for P, including Mongolian variation selectors
and U+18A9).
Note that the "otherwise" case is the only reliable way to give an explicitly renderedhyphen, e.g. in texts about hyphenation, for the Armenian and Mongolian hyphens.
If the line LV broken at the point indicated, the soft hyphen is rendered visibly; before
the line break in the case of SHY and ASHY, after the line break in the case of MSHY.
By default there is QR spell change if the line is broken.
For some orthographies the default rules may need to be modified slightly, as longas no dictionary or heuristics dependency is introduced in the interpretation of explicit
soft hyphens. E.g., for "old" orthography German, for the particular case of ‘ck’ the rules
for the line break case would be augmented by c<SHY>k -> k-/k and C<SHY>K -> K-/K,
where / indicates the line break, but otherwise stay with the default of no spell changeand visible rendering before the line break.
For example ELOMHWW<SHY>WnQJ is rendered either as ELOMHWWnQJ or ELOMHWW-/WnQJ,6FKLII<SHY>)DKUW either as 6FKLIIDKUW or 6FKLII-/)DKUW, 6DXHUVWRI<ZWJ>I<SHY>)ODVFKH isrendered either as 6DXHUVWRIIIODVFKH or as 6DXHUVWRII-/)ODVFKH, and for "old" orthography
German 6XF<SHY>NHU should be rendered either as 6XFNHU or 6XN-/NHU.

3