Configurable character variant unification

A method for configurable character variant unification is provided in the illustrative embodiments. A determination is made that a unification profile is applicable to a circumstance in which a character variant has been selected. The character variant is a variation of a character in a set of variations of the character such that each variation of the character in the set is represented by a unique Unicode code point. A unification repository is identified according to the profile. A determination is made whether the character variant satisfies a unification rule. Responsive to the character variant not satisfying the unification rule, a different variation of the character is selected from the unification repository, the different variation forming a replacement character variant. The replacement character variant is used in place of the character variant.

TECHNICAL FIELD

The present invention relates generally to a method for providing consistent computer input in multiple languages. More particularly, the present invention relates to a method for configurable unification of character variants received from data input.

BACKGROUND

There are alphabet and non-alphabet languages in the world. For example, Chinese, Japanese and Korean borrowed alphabetic elements to represent their own phonetic symbols or strokes.

A computer keyboard is a common device for providing a computer input. A keyboard is language-specific such that the alphabet or non-alphabet keys available on the keyboard can be pressed to directly input only those characters or symbols in the keyboard's language that are assigned to those keys. For inputting other characters or symbols in the language, a user may need to press a combination of keys on the keyboard to invoke a specific input method application for the language.

Many languages have sets of characters or symbols (e.g., character alphabet in English, or phonetic or stroke alphabet in other languages) that are too large to accommodate on a keyboard. Many languages need other ways of mapping the keyboard keys to the characters or symbols in the language's set of characters or symbols. Using the keyboard keys according to the mapping produces the mapped characters or symbols in the language. Furthermore, the phonetic or stroke alphabets of many languages do not use characters to form words in the manner of the English language, but have a single character or collection of characters that represent words. Thus, providing computer input in many languages is not as simple as pressing the letter-keys on the keyboard but an indirect process of pressing a combination of keys to generate characters not available as keys on the keyboard.

Unicode is a method of coding characters of multiple languages. A Unicode table comprises unique codes called code points assigned to characters of one or more languages. A code point comprises an alphanumeric representation that can be generated on commonly used keyboard configurations, such as an English language QWERTY keyboard.

To enter a code point, the user generally supplies an indication that the alphanumeric string following the indication is a Unicode code point as is to be translated using a Unicode table to generate a character. For example, using a QWERTY keyboard, the user presses the ALT key, keeps the ALT key depressed while entering the code point, and releases the ALT key when the code point entry is complete.

An application called a Unicode input method application (hereinafter, “input method”, or “UIM”) intercepts the Unicode code point that the user enters. A Unicode editor is an example UIM. The UIM looks up a Unicode table to find the character that matches the code point that the user entered. The UIM supplies the character to a target application to which the user is supplying the input.

Different sections in a Unicode table comprise different unique sets of unique code points to represent different sets of characters in different languages. In other words, a code point in all of Unicode is unique to a specific character in a specific language.

SUMMARY

The illustrative embodiments provide a method, for configurable character variant unification. An embodiment includes a method for configurable character variant unification. The embodiment determines that a unification profile is applicable to a circumstance in which a character variant has been selected, wherein the character variant is a variation of a character in a set of variations of the character such that each variation of the character in the set is represented by a unique Unicode code point. The embodiment identifies a unification repository according to the profile. The embodiment determines whether the character variant satisfies a unification rule. The embodiment selects, responsive to the character variant not satisfying the unification rule, a different variation of the character from the unification repository, the different variation forming a replacement character variant. The embodiment uses the replacement character variant in place of the character variant.

Another embodiment includes a computer usable program product comprising a computer readable storage device including computer usable code for configurable character variant unification. The embodiment further includes computer usable code for determining that a unification profile is applicable to a circumstance in which a character variant has been selected, wherein the character variant is a variation of a character in a set of variations of the character such that each variation of the character in the set is represented by a unique Unicode code point. The embodiment further includes computer usable code for identifying a unification repository according to the profile. The embodiment further includes computer usable code for determining whether the character variant satisfies a unification rule. The embodiment further includes computer usable code for selecting, responsive to the character variant not satisfying the unification rule, a different variation of the character from the unification repository, the different variation forming a replacement character variant. The embodiment further includes computer usable code for using the replacement character variant in place of the character variant.

Another embodiment includes a data processing system for configurable character variant unification. The embodiment further includes a storage device including a storage medium, wherein the storage device stores computer usable program code. The embodiment further includes a processor, wherein the processor executes the computer usable program code. The embodiment further includes computer usable code for determining that a unification profile is applicable to a circumstance in which a character variant has been selected, wherein the character variant is a variation of a character in a set of variations of the character such that each variation of the character in the set is represented by a unique Unicode code point. The embodiment further includes computer usable code for identifying a unification repository according to the profile. The embodiment further includes computer usable code for determining whether the character variant satisfies a unification rule. The embodiment further includes computer usable code for selecting, responsive to the character variant not satisfying the unification rule, a different variation of the character from the unification repository, the different variation forming a replacement character variant. The embodiment further includes computer usable code for using the replacement character variant in place of the character variant.

DETAILED DESCRIPTION

A character variant is a variation of a character within a language or across different languages. For example, a first type of character variants involves different characters in a given language or across different languages, where the different characters look different but are pronounced in a similar manner, convey similar meanings, or both. This type of variants is referred to hereinafter as ‘distinct variants’. For example, in the Chinese language, this type of character variants can be found between simplified Chinese and traditional Chinese. The simplified character is visually different from the traditional character, but the simplified and the traditional characters are pronounced approximately the same and convey approximately the same meanings.

A second type of character variants is called ‘resemblance variants’. Resemblance variants are different characters in a given language or across different languages, where the different characters look similar, but may be pronounced in a similar manner or different manners, may convey similar meanings or different meanings, or some combination thereof. Often, resemblance variants have their origin in a character in one language, e.g., Chinese, where the character was adopted by other languages, e.g., Japanese or Korean, and gradually became a part of those languages without a change of the written expression of the character.

As a result, the original character and their resemblance variants were all added in the Unicode tables that contain characters from different languages, each character and its resemblance variants having a distinct code point and being treated as a unique character by computers although they visually look alike. A user can generally input all the character variants by using a UIM or other input methods.

The illustrative embodiments recognize a particular problem with resemblance variants. For example, if a user searches for a character, and if the character has resemblance variants, the user is presented with the resemblance variants. For example, if the user is searching for character302inFIG. 3, the user may be presented with characters302,304, and306, where characters304, and306are resemblance variants of a common character, e.g., of character302.

The illustrative embodiments recognize that when faced with resemblance variants, a user may unintentionally, unknowingly, or even maliciously select a different resemblance variant than the variant that was intended. The illustrative embodiments recognize that selecting or entering a different resemblance variant of a character than an intended resemblance variant of the character can pose a variety of problems in data management.

For example, a particular variant may not be allowed in information processing in certain languages or certain regions. Entering a prohibited variant can therefore cause errors, costs, and delays in processing of the information in which the prohibited variant is included.

As another example, a user may be looking for the character to enter as a user ID, password, filename, or other phrases during the information processing. Selecting the incorrect variant can cause login error, login lockout, security flagging, existing file not being found, new file being created with a name that will not be found by others, and generally data being created or manipulated in an inconsistent or erroneous fashion.

As another example, suppose the user is engaged in a record manipulation operation in a database. Using wrong, inconsistent, or different variants can result in ghost records being created in the database. Using wrong, inconsistent, or different variants can also increase the data processing time and resource usage due to the extra effort needed to process the different variants, e.g., by employing different language processing tools for processing the variants.

The illustrative embodiments further recognize that not only do resemblance variants have the potential to cause data processing complications, they also have the potential to cause social and cross-cultural issues and insensitivities. Thus, the illustrative embodiments recognize that unintended, accidental, or malicious misuse of resemblance variants can adversely affect the quality of data where used, and also have social, geographical, political, and economic consequences.

The illustrative embodiments used to describe the invention generally address and solve the above-described problems and other problems related to using character variants. The illustrative embodiments provide a method for configurable character variant unification.

Unification is the process of unifying one or more character variants back to a common character. A unification database (database, databases) according to an embodiment is a repository of variants that can be unified to a character in a given language. For example, a unification database for simplified Chinese language includes a list of characters, their corresponding code points in simplified Chinese, and their respective distinct variants, resemblance variants, or both. Any number of unification databases can be created for any number of languages without limitation within the scope of the illustrative embodiments.

Furthermore, more than one unification databases may exist for a given language. A custom unification database that includes entries for characters in multiple languages and their variants from a combination of languages is also contemplated within the scope of the illustrative embodiments. The unification database can take any suitable form, including but not limited to a relational database, a flat-file, an XML file, an index file, a spreadsheet, a table, and the like.

A unification rule (rule, rules) is logic in any suitable form to resolve a variant to an intended character or a different variant using one or more unification databases. For example, given a character variant selected by a user, an embodiment uses a unification rule to select a suitable changed variant of the character from a unification database.

Any number of unification rules is permissible without departing the scope of the illustrative embodiments. A set of unification rules according to the illustrative embodiments can include unification rules for any combination of different languages, different geographical regions, different locales, and different contexts of usage. Furthermore, different unification rules may produce different changed variants depending on various considerations. For example, if the user selects a variant at login time, for use in a user ID, an embodiment uses a different unification rule according to the login context as compared to the unification rule used when the user selects a variant to embed in a document. The different unification rules may use the same or different one or more unification databases, and produce same or different changed variants under the different contexts.

A unification profile (profile, profiles) comprises one or more unification levels (level, levels). A unification level defines how and which unification databases are to be combined for a particular unification exercise. A unification profile applies to a user, a group of users, a document, an application, a data storage, a locale, a geographical region, or some combination thereof.

For example, a user can define four example levels—Simplified Chinese only (level 1); Traditional Chinese only (level 2); Simplified Chinese and Traditional Chinese (level 3); and Simplified Chinese, Traditional Chinese, and Japanese Kanji (level 4). The user can further define, for example, that different levels apply to different users, groups, applications, or storage, in a manner that further narrow or define the application of the profile.

For example, a profile might apply to a group and a level therein might apply to a specific user in that group. As another example, a profile might apply to a geographical region and a level therein might apply to a specific group operating in that region. These example ways of constructing profiles and levels, and example applications of the profiles and levels are not intended to be limiting on the illustrative embodiments. From this disclosure, those of ordinary skill in the art will be able to conceive many other ways of constructing and using profiles and levels, and the same are contemplated within the scope of the illustrative embodiments.

In operation, when a user is going to be using character variants, an embodiment selects a profile and a level in the profile according to which the embodiment will unify the variants to produce an output variant for the user-select variant. In one embodiment, the user selects the profile and the level. In another embodiment, the profile and level are selected on behalf of the user, such as by an administrator. In another embodiment, a policy determines the profile and level that the embodiment should use.

Once the profile and the level are selected, an embodiment performs the variant unification on a character variant selected by the user to produce an output character variant. The embodiment uses the one or more unification database(s) corresponding to the selected profile and level, according to one or more unification rules governing the circumstances of the selection of the character variant by the user.

An embodiment allows a user, an administrator, or both to create, modify, or manipulate a unification profile, a unification level within a unification profile, a unification rule, a unification database, or some combination thereof. For example, one embodiment allows a user, whose variant selections are to be unified, to manipulate a level but not the profile itself, and allows an administrator to manipulate the profile. Another example embodiment allows the user create or manipulate entries in a unification database but only allows an administrator to manipulate unification rules. Another example embodiment allows different users to manipulate different profiles, levels, rules, databases, or a combination thereof.

A method of an embodiment described herein, when implemented to execute on a data processing system, comprises substantial advancement of the functionality of that data processing system. For example, an embodiment enables the data processing system to identify and unify particular variants that may not be allowed in information processing in certain languages or certain regions, prior to such variants entering such information processing. Such identification and unification ability is unavailable in presently operating data processing systems. Thus, a substantial advancement of such data processing systems by executing a method of an embodiment comprises the prevention or mitigation of the errors, costs, and delays in processing of the information caused by the prior art data processing systems allowing the entry and storing of undesirable character variants.

The illustrative embodiments are described with respect to certain languages, characters, character variants, documents, identifiers, contexts, profiles, levels, databases, repositories, policies, logic, rules, data processing systems, environments, components, and applications only as examples. Any specific manifestations of such artifacts are not intended to be limiting to the invention. Any suitable manifestation of these and other similar artifacts can be selected within the scope of the illustrative embodiments.

FIG. 1depicts a block diagram of a network of data processing systems in which illustrative embodiments may be implemented. Data processing environment100is a network of computers in which the illustrative embodiments may be implemented. Data processing environment100includes network102. Network102is the medium used to provide communications links between various devices and computers connected together within data processing environment100. Network102may include connections, such as wire, wireless communication links, or fiber optic cables. Server104and server106couple to network102along with storage unit108. Software applications may execute on any computer in data processing environment100.

In addition, clients110,112, and114couple to network102. A data processing system, such as server104or106, or client110,112, or114may contain data and may have software applications or software tools executing thereon.

Only as an example, and without implying any limitation to such architecture,FIG. 1depicts certain components that are usable in an example implementation of an embodiment. For example, servers104and106, and clients110,112,114, are depicted as servers and clients only as example and not to imply a limitation to a client-server architecture. As another example, an embodiment can be distributed across several data processing systems and a data network as shown, whereas another embodiment can be implemented on a single data processing system within the scope of the illustrative embodiments.

Input method application103is any suitable UIM as described herein. Application105implements an embodiment described herein. Unification rules107are a set of one or more unification rules usable in an embodiment. Unification databases109are a set of one or more unification repositories of any suitable types as described herein. Unification profiles111are a set of one or more unification profiles usable in an embodiment.

With reference toFIG. 2, this figure depicts a block diagram of a data processing system in which illustrative embodiments may be implemented. Data processing system200is an example of a computer, such as servers104and106, or clients110,112, and114inFIG. 1, or another type of device in which computer usable program code or instructions implementing the processes may be located for the illustrative embodiments. Data processing system200is also representative of other devices in which computer usable program code or instructions implementing the processes of the illustrative embodiments may be located. Data processing system200is described as a computer only as an example, without being limited thereto. Implementations in the form of other devices may modify data processing system200and even eliminate certain depicted components there from without departing from the general description of the operations and functions of data processing system200described herein.

Instructions for the operating system, the object-oriented programming system, and applications or programs, such as input method application103, application105, and unification rules107inFIG. 1, are located on storage devices, such as hard disk drive226, and may be loaded into at least one of one or more memories, such as main memory208, for execution by processing unit206. The processes of the illustrative embodiments may be performed by processing unit206using computer implemented instructions, which may be located in a memory, such as, for example, main memory208, read only memory224, or in one or more peripheral devices.

With reference toFIG. 3, this figure depicts a table of example resemblance variants of an example character that can be configurably unified in accordance with an illustrative embodiment. Application105inFIG. 1can be used to unify variants302,304, and306into the variant302.

Table300shows that Han character302is an original variant that corresponds to Unicode code point U+5317, and has at least two other resemblance variants304and306, which correspond to code points U+F963 and U+2f82b, respectively. For example, if a user uses a PINYIN, a commonly used Chinese phonetic input method editor, and enters “bei” using a keyboard, the user is likely to be presented with all three variants from which the user selects one variant to use.

Suppose a given usage context or circumstance requires Han characters but the user selects variant304corresponding to code point U+F963, thereby selecting a Chinese-Japanese-Korean (CJK) compatibility ideograph instead. Such a selection would ordinarily cause an error or other complication in the processing of the data including the selected variant. If character variant unification feature is enabled, such as by using an application implementing an embodiment, e.g., application105inFIG. 1, the unification feature unifies selected variant304to produce variant302as the output.

An entry in a unification database in databases109ofFIG. 1establishes the correspondence between variants302,304, and306. A unification rule in rules107ofFIG. 1allows the application to determine that under the circumstances of the selection, variant304should be changed to variant302. Only as an example to illustrate the operation of unification rules, and not to imply any limitation on the illustrative embodiments, another unification rule in rules107ofFIG. 1may allow the application to determine that under different circumstances of the selection, variant304should be changed to variant306, or variant304should be accepted as the correct variant.

With reference toFIG. 4, this figure depicts a block diagram of a configuration for configurable character variant unification in accordance with an illustrative embodiment. Application402can be implemented using application105inFIG. 1.

User404uses input device406to enter a search for a character. UIM408presents a set of characters from which the user selects character410. UIM408produces code point412of selected character410. Code point412serves as an input to application402.

Component414allows a user or administrator to define one or more unification profiles, one or more unification levels within a unification profile, one or more unification rules, one or more unification databases, or a combination thereof. Component416selects a unification profile, e.g., unification profile418and a level therein, and one or more unification databases according to profile418, e.g., unification database420.

Component422applies one or more unification rules424to code point412according to the selected level in profile418. Component412produces output character variant426, or a Unicode code point corresponding thereto.

With reference toFIG. 5, this figure depicts a flowchart of an example process for configurable character variant unification in accordance with an illustrative embodiment. Process500can be implemented in application402inFIG. 4.

The application receives a Unicode code point value of a selected character variant from a UIM (block502). The application determines a unification profile that is active or applicable for unifying the selected character variant and a unification level in that profile (block504).

For example, according to one embodiment, the profile and level are selected by a user or administrator and the application uses the selected profile and level in block504. According to another embodiment, the application determines a circumstance of the selected character variant, e.g., the user's identification, the user's membership in a group, the document where the selected character variant is going to be used, a context in which the variant has been selected, an application that is to receive the character from the user, other circumstances, or a combination thereof. Based on the circumstance of usage, the application selects a suitable profile and a level therein for use in block504.

The application selects one or more unification databases according to the selected profile and level (block506). The application applies one or more unification rules on the selected character variant using the selected unification databases of block506(block508).

From applying a unification rule, the application determines whether the selected character variant complies with the rule (block510). If the selected character variant is the correct variant (Yes” path of block510), the application sends the selected character variant to the target application that was to receive the character (block512). The application ends process500thereafter.

If the selected character variant is the correct variant (Yes” path of block510), the application replaces the Unicode code point received in block502with a Unicode code point of another character variant identified in a unification database according to a unification rule (block514). The application sends the replaced character variant to the target application in block514and ends process500thereafter.

With reference toFIG. 6, this figure depicts a flowchart of an example process for configuring character variant unification in accordance with an illustrative embodiment. Process600can be implemented in application402inFIG. 4.

The application causes to be created, or creates, a unification profile that is applicable to a circumstance in which a character variant might be selected (block602). The application causes to be defined, or defines, a unification level in the unification profile of block602(block604).

For a level in the profile, the application associates one or more unification databases with the profile level (block606). The application enables the profile and/or level to be used with a set of unification rules (block608). For example, an administrator may desire to create some profiles and/or levels in reserve without enabling them for use. Similarly, an administrator may enable or disable a profile or a level within the profile according to changing needs for character variant unification.

The application repeats blocks604-608for as many levels as may be desired in a profile. The application repeats blocks602-608for as many profiles with as many levels as may be desired in a given implementation. The application ends process600thereafter.

Thus, a computer implemented method is provided in the illustrative embodiments for configurable character variant unification.