Data processing method and device

A data processing method and device are provided, where the method includes encoding data by using an arithmetic coding algorithm, so as to obtain a code value interval. When a code value corresponding to the data exists in the code value interval, obtaining the code value according to the code value interval. The method further includes comparing a quantity of bits of the code value with a quantity of bits of the data, so as to obtain a comparison result, and performing a storage operation according to the comparison result.

TECHNICAL FIELD

The present invention relates to the field of data processing, and in particular, to a data processing method and device.

BACKGROUND

In recent years, arithmetic coding is another extremely practical lossless compression algorithm. A core idea of the arithmetic coding is that all encoded symbols that may appear in source data are mapped to an integer set, and a corresponding appearance probability is assigned to each encoded symbol (it is required that a sum of appearance probabilities of all characters is 1 herein). Each character occupies a half-open and half-closed consecutive interval in an interval [0, 1] according to the appearance probability of each character, an interval length value is a probability value, and intervals are mutually independent. A string that needs to be encoded is mapped into an integer sequence according to a mapping table. The source data is gradually converted, according to appearance probabilities of the encoded symbols in the source data that needs to be encoded, into a real number interval corresponding to the interval [0, 1]. A real number in the interval is used as a code value and is saved in a computer. An interval for performing encoding next time is an interval obtained by means of encoding previous time. Appearance probability ratios of all the symbols remain the same each time. During decoding, the binary code value is resaved to the corresponding integer sequence according to inverse conversion, and then the integer sequence is mapped to the original string. For example, for integer set space {0, 1, 2, 3}, appearance probability distribution is {0.2, 0.5, 0.2, 0.1}. In this case, corresponding to data whose input sequence is <210013>, encoding intervals are sequentially [0.7, 0.9], [0.74, 0.84], [0.74, 0.76], [0.74, 0.744], [0.7408, 0.7428], and [0.7426, 0.7428]. Finally, a code value interval corresponding to the data is [0.7426, 0.7428] (the encoding interval corresponding to the last character sequence), and a code value of the data is a value in [0.7426, 0.7428].

For to-be-encoded data, in existing arithmetic coding, data is directly compressed without considering whether there is a compression gain, and then a corresponding code value obtained after the arithmetic coding is saved. However, data storage space is increased in the prior art because a quantity of bits of a code value corresponding to some data is large.

SUMMARY

Embodiments of the present invention provide a data processing method and device, and the method can reduce data storage space.

According to a first aspect, a data processing method is provided, including encoding data by using an arithmetic coding algorithm, so as to obtain a code value interval; when a code value corresponding to the data exists in the code value interval, obtaining the code value according to the code value interval; comparing a quantity of bits of the code value with a quantity of bits of the data, so as to obtain a comparison result; and performing a storage operation according to the comparison result.

With reference to the first aspect, in a first possible implementation manner, the comparison result is that the quantity of bits of the code value is less than the quantity of bits of the data, and the performing a storage operation according to the comparison result includes: storing the code value according to the comparison result.

With reference to the first aspect, in a second possible implementation manner, the comparison result is that the quantity of bits of the code value is greater than or equal to the quantity of bits of the data, and the performing a storage operation according to the comparison result includes: storing the data according to the comparison result.

With reference to the first possible implementation manner, in a third possible implementation manner, the method further includes: performing an application operation of the data according to the code value, where the application operation includes at least one of an equivalent comparison, sorting, or a fuzzy search.

With reference to the third possible implementation manner, in a fourth possible implementation manner, the data is an identity ID type string, the application operation includes the equivalent comparison, and the performing an application operation of the data according to the code value includes: when the code value is equal to a to-be-compared code value, determining that the data and data corresponding to the to-be-compared code value are the same data.

With reference to the third possible implementation manner, in a fifth possible implementation manner, the data is an ID type string or a letter string of a field, the application operation includes sorting, and the performing an application operation of the data according to the code value includes: determining a location of the code value in a to-be-sorted code value according to the code value, where the location of the code value is used to represent a location of the data in data corresponding to the to-be-sorted code value.

With reference to the third possible implementation manner, in a sixth possible implementation manner, the application operation includes the fuzzy search, and the performing an application operation of the data according to the code value includes: determining, according to whether the code value falls within an encoding interval of a prefix string that needs to be fuzzily searched, whether the data includes the prefix string, and when the code value falls within the encoding interval of the prefix string that needs to be fuzzily searched, the data includes the prefix string, or when the code value falls outside the encoding interval of the prefix string that needs to be fuzzily searched, the data does not include the prefix string.

With reference to any one of the first aspect or the first to the sixth possible implementation manners, in a seventh possible implementation manner, the encoding data by using an arithmetic coding algorithm, so as to obtain a code value interval includes: encoding the data by using the arithmetic coding algorithm, so as to obtain an encoding interval; re-extending the encoding interval of the data, so as to obtain a re-extended encoding interval; and continuing to encode the data according to the re-extended encoding interval by using the arithmetic coding algorithm, so as to obtain the code value interval.

With reference to the seventh possible implementation manner, in an eighth possible implementation manner, the re-extending the encoding interval of the data, so as to obtain a re-extended encoding interval includes: when at least one of the following conditions is met, re-extending the encoding interval of the data, so as to obtain the re-extended encoding interval and record a location of a corresponding re-extended character: a quantity of encoded characters of the data reaches a character quantity threshold or a length of the encoding interval of the data is less than an interval threshold.

With reference to any one of the first aspect or the first to the eighth possible implementation manners, in a ninth possible implementation manner, when a code value corresponding to the data exists in the code value interval, before the obtaining the code value according to the code value interval, the method further includes: determining whether an appropriate code value corresponding to the data exists in the code value interval.

With reference to the ninth possible implementation manner, in a tenth possible implementation manner, the method further includes: storing the data when no appropriate code value corresponding to the data exists in the code value interval.

According to a second aspect, a data processing method is provided, including: encoding data by using an arithmetic coding algorithm, so as to obtain an encoding interval; re-extending the encoding interval of the data, so as to obtain a re-extended encoding interval; continuing to encode the data according to the re-extended encoding interval by using the arithmetic coding algorithm, so as to obtain a code value interval; obtaining the code value according to the code value interval; and storing the code value.

With reference to the second aspect, in a first possible implementation manner, the re-extending the encoding interval of the data, so as to obtain a re-extended encoding interval includes: when at least one of the following conditions is met, re-extending the encoding interval of the data, so as to obtain the re-extended encoding interval and record a location of a corresponding re-extended character: a quantity of encoded characters of the data reaches a character quantity threshold or a length of the encoding interval of the data is less than an interval threshold.

According to a third aspect, a data processing method is provided, including: obtaining a code value of data and a location of a re-extended character; decoding the code value of the data by using an arithmetic coding algorithm, so as to obtain a decoding interval; re-scaling the decoding interval of the data according to the location of the re-extended character, so as to obtain a re-scaled decoding interval; and continuing to decode the data according to the re-scaled decoding interval by using the arithmetic coding algorithm, so as to obtain the data.

With reference to the third aspect, in a first possible implementation manner, the re-scaling the decoding interval of the data according to the location of the re-extended character, so as to obtain a re-scaled decoding interval includes: determining a location of a re-scaled character according to the location of the re-extended character, where the location of the re-extended character is inverse to the location of the re-scaled character; and re-scaling the decoding interval of the data according to the location of the re-scaled character, so as to obtain the re-scaled decoding interval.

According to a fourth aspect, a data processing method is provided, including: encoding data by using an arithmetic coding algorithm, so as to obtain a code value interval; obtaining the code value according to the code value interval; storing the code value; and performing an application operation of the data according to the code value, where the application operation includes at least one of an equivalent comparison, sorting, or a fuzzy search.

With reference to the fourth aspect, in a first possible implementation manner, the data is an ID type string, the application operation includes the equivalent comparison, and the performing an application operation of the data according to the code value includes: when the code value is equal to a to-be-compared code value, determining that the data and data corresponding to the to-be-compared code value are the same data.

With reference to the fourth aspect, in a second possible implementation manner, the data is an ID type string or a letter string of a field, the application operation includes sorting, and the performing an application operation of the data according to the code value includes: determining a location of the code value in a to-be-sorted code value according to the code value, where the location of the code value is used to represent a location of the data in data corresponding to the to-be-sorted code value.

With reference to the fourth aspect, in a third possible implementation manner, the application operation includes the fuzzy search, and the performing an application operation of the data according to the code value includes: determining, according to whether the code value falls within an encoding interval of a prefix string that needs to be fuzzily searched, whether the data includes the prefix string; and when the code value falls within the encoding interval of the prefix string that needs to be fuzzily searched, the data includes the prefix string, or when the code value falls outside the encoding interval of the prefix string that needs to be fuzzily searched, the data does not include the prefix string.

According to a fifth aspect, a data processing device is provided, including: an encoding unit, configured to encode data by using an arithmetic coding algorithm, so as to obtain a code value interval; an obtaining unit, configured to obtain, when a code value corresponding to the data exists in the code value interval, the code value according to the code value interval; a comparison unit, configured to compare a quantity of bits of the code value with a quantity of bits of the data, so as to obtain a comparison result; and a first storage unit, configured to perform a storage operation according to the comparison result.

With reference to the fifth aspect, in a first possible implementation manner, the comparison result is that the quantity of bits of the code value is less than the quantity of bits of the data, and the first storage unit saves the code value according to the comparison result.

With reference to the fifth aspect, in a second possible implementation manner, the comparison result is that the quantity of bits of the code value is greater than or equal to the quantity of bits of the data, and the first storage unit saves the data according to the comparison result.

With reference to the first possible implementation manner of the fifth aspect, in a third possible implementation manner, the device further includes an application unit, configured to perform an application operation of the data according to the code value, where the application operation includes at least one of an equivalent comparison, sorting, or a fuzzy search.

With reference to the third possible implementation manner of the fifth aspect, in a fourth possible implementation manner, the data is an ID type string, the application operation includes the equivalent comparison, and when the code value is equal to a to-be-compared code value, the application unit determines that the data and data corresponding to the to-be-compared code value are the same data.

With reference to the third possible implementation manner of the fifth aspect, in a sixth possible implementation manner, the data is an ID type string or a letter string of a field, the application operation includes sorting, and the application unit determines a location of the code value in a to-be-sorted code value according to the code value, where the location of the code value is used to represent a location of the data in data corresponding to the to-be-sorted code value.

With reference to the third possible implementation manner of the fifth aspect, in an eighth possible implementation manner, the application operation includes the fuzzy search, and the application unit determines, according to whether the code value falls within an encoding interval of a prefix string that needs to be fuzzily searched, whether the data includes the prefix string; and when the code value falls within the encoding interval of the prefix string that needs to be fuzzily searched, the data includes the prefix string, or when the code value falls outside the encoding interval of the prefix string that needs to be fuzzily searched, the data does not include the prefix string.

With reference to any one of the fifth aspect or the first to the ninth possible implementation manners of the fifth aspect, in a tenth possible implementation manner, the encoding unit encodes the data by using the arithmetic coding algorithm, so as to obtain an encoding interval; re-extends the encoding interval of the data, so as to obtain a re-extended encoding interval; and continues to encode the data according to the re-extended encoding interval by using the arithmetic coding algorithm, so as to obtain the code value interval.

With reference to the tenth possible implementation manner of the fifth aspect, in an eleventh possible implementation manner, when at least one of the following conditions is met, the encoding unit re-extends the encoding interval of the data, so as to obtain the re-extended encoding interval and record a location of a corresponding re-extended character: a quantity of encoded characters of the data reaches a character quantity threshold or a length of the encoding interval of the data is less than an interval threshold.

With reference to any one of the fifth aspect or the first to the eleventh possible implementation manners of the fifth aspect, in a twelfth possible implementation manner, the device further includes a determining unit, configured to determine, before the obtaining unit obtains the code value, whether an appropriate code value corresponding to the data exists in the code value interval.

With reference to the twelfth possible implementation manner of the fifth aspect, in a thirteenth possible implementation manner, the device further includes a second storage unit, configured to save the data when no appropriate code value corresponding to the data exists in the code value interval.

According to a sixth aspect, a data processing device is provided, including: a first encoding unit, configured to encode data by using an arithmetic coding algorithm, so as to obtain an encoding interval; an extension unit, configured to re-extend the encoding interval of the data, so as to obtain a re-extended encoding interval; a second encoding unit, configured to continue to encode the data according to the re-extended encoding interval by using the arithmetic coding algorithm, so as to obtain a code value interval; an obtaining unit, configured to obtain a code value according to the code value interval; and a storage unit, configured to save the code value.

With reference to the sixth aspect, in a first possible implementation manner, when at least one of the following conditions is met, the extension unit re-extends the encoding interval of the data, so as to obtain the re-extended encoding interval and record a location of a corresponding re-extended character: a quantity of encoded characters of the data reaches a character quantity threshold or a length of the encoding interval of the data is less than an interval threshold.

According to a seventh aspect, a data processing device is provided, including a first obtaining unit, configured to obtain a code value of data and a location of a re-extended character, a first decoding unit, configured to decode the code value of the data by using an arithmetic coding algorithm, so as to obtain a decoding interval, a scaling unit, configured to re-scale the decoding interval of the data according to the location of the re-extended character, so as to obtain a re-scaled decoding interval, and a second decoding unit, configured to continue to decode the data according to the re-scaled decoding interval by using the arithmetic coding algorithm, so as to obtain the data.

With reference to the seventh aspect, in a first possible implementation manner, the scaling unit determines a location of a re-scaled character according to the location of the re-extended character, where the location of the re-extended character is inverse to the location of the re-scaled character; and re-scales the decoding interval of the data according to the location of the re-scaled character, so as to obtain the re-scaled decoding interval.

According to an eighth aspect, a data processing device is provided, including an encoding unit, configured to encode data by using an arithmetic coding algorithm, so as to obtain a code value interval, an obtaining unit, configured to obtain a code value according to the code value interval; a storage unit, configured to save the code value, and an application unit, configured to perform an application operation of the data according to the code value, where the application operation includes at least one of an equivalent comparison, sorting, or a fuzzy search.

With reference to the eighth aspect, in a first possible implementation manner, the data is an ID type string, the application operation includes the equivalent comparison, and when the code value is equal to a to-be-compared code value, the application unit determines that the data and data corresponding to the to-be-compared code value are the same data.

With reference to the eighth aspect, in a third possible implementation manner, the data is an ID type string or a letter string of a field, the application operation includes sorting, and the application unit determines a location of the code value in a to-be-sorted code value according to the code value, where the location of the code value is used to represent a location of the data in data corresponding to the to-be-sorted code value.

With reference to the eighth aspect, in a fifth possible implementation manner, the application operation includes the fuzzy search, and the application unit determines, according to whether the code value falls within an encoding interval of a prefix string that needs to be fuzzily searched, whether the data includes the prefix string; and when the code value falls within the encoding interval of the prefix string that needs to be fuzzily searched, the data includes the prefix string, or when the code value falls outside the encoding interval of the prefix string that needs to be fuzzily searched, the data does not include the prefix string.

Based on the foregoing technical solutions, in the embodiments of the present invention, data is encoded by using an arithmetic coding algorithm, so as to obtain a code value interval. When a code value corresponding to the data exists in the code value interval, the code value is obtained according to the code value interval; a quantity of bits of the code value is compared with a quantity of bits of the data, so as to obtain a comparison result; and a storage operation is performed according to the comparison result. In the embodiments of the present invention, data storage space can be reduced.

DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

FIG. 1is a schematic flowchart of a data compression method according to an embodiment of the present invention. The method inFIG. 1may be executed by a data compression device inFIG. 4. The method inFIG. 1includes the following steps.

110. Encode data by using an arithmetic coding algorithm, so as to obtain a code value interval.

Specifically, the code value interval may be an encoding interval corresponding to the last string of the data.

It should be understood that the data in this embodiment of the present invention may be an identity (Identity, ID) type string or a letter string of a field, or the like. The ID type string may include a data document number, an index number of a book, a bus license plate number, a product document number, an international mobile subscriber identification (International Mobile Subscriber Identification, IMSI) number, or an international mobile equipment identity (International Mobile Equipment Identity, IMEI), or the like. The letter string may include phonetic symbols corresponding to Chinese, Korean, or Japanese, or the like.

120. When a code value corresponding to the data exists in the code value interval, obtain the code value according to the code value interval.

In other words, when an appropriate code value may be obtained from the code value interval, the code value is obtained according to the code value interval. For example, the appropriate code value may be a code value that meets a particular quantity of bits, for example, a quantity of binary bits of the code value does not exceed 16 bits, 32 bits, or 64 bits.

It should be noted that the code value may be obtained according to the code value interval and an existing method, or the code value may be obtained according to a preset condition. For example, the preset condition is that the quantity of the binary bits of the code value does not exceed 16 bits, 32 bits, 64 bits, or the like. The preset condition is not limited in this embodiment of the present invention.

130. Compare a quantity of bits of the code value with a quantity of bits of the data, so as to obtain a comparison result.

Specifically, the comparison result may include that the quantity of bits of the code value is less than the quantity of bits of the data, the quantity of bits of the code value is equal to the quantity of bits of the data, or the quantity of bits of the code value is greater than the quantity of bits of the data.

140. Perform a storage operation according to the comparison result.

Specifically, data is encoded by using an arithmetic coding algorithm, so as to obtain a code value interval, then it is determined whether an appropriate code value may be obtained from the code value interval (an encoding interval corresponding to the last string of the data), and if no appropriate code value may be obtained, the data is directly saved without using arithmetic coding. If an appropriate code value may be obtained, a quantity of bits of the code value is compared with a quantity of bits of the data (a compression gain is determined). Specifically, it is determined whether the quantity of bits of the appropriate code value is greater than or equal to the quantity of bits of the source data, and if the quantity of bits of the appropriate code value is greater than or equal to the quantity of bits of the source data, that is, there is no compression gain, arithmetic coding is abandoned and the data may be directly saved. If the quantity of bits of the appropriate code value is less than the quantity of bits of the source data, that is, there is a compression gain, the code value is saved.

Therefore, in this embodiment of the present invention, data is encoded by using an arithmetic coding algorithm, so as to obtain a code value interval. When a code value corresponding to the data exists in the code value interval, the code value is obtained according to the code value interval; a quantity of bits of the code value is compared with a quantity of bits of the data, so as to obtain a comparison result; and a storage operation is performed according to the comparison result. In this embodiment of the present invention, incorrect encoding can be reduced, and data storage space can be reduced.

Optionally, in another embodiment, the comparison result is that the quantity of bits of the code value is less than the quantity of bits of the data, and the code value is saved according to the comparison result in step140.

Specifically, when the quantity of bits of the code value is less than the quantity of bits of the data, it indicates that a compression gain exists, and the code value is saved.

Alternatively, in another embodiment, the comparison result is that the quantity of bits of the code value is greater than or equal to the quantity of bits of the data, and the data is saved according to the comparison result in step140.

Specifically, when the quantity of bits of the code value is greater than or equal to the quantity of bits of the data, it indicates that there is no compression gain, and the data is saved.

Optionally, in another embodiment, when the code value is saved, the method in this embodiment of the present invention further includes performing an application operation of the data according to the code value. The application operation includes at least one of an equivalent comparison, sorting, or a fuzzy search.

Specifically, data is encoded by using an arithmetic coding algorithm, so as to obtain a code value interval; when a code value corresponding to the data exists in the code value interval, the code value is obtained according to the code value interval; a quantity of bits of the code value is compared with a quantity of bits of the data, so as to obtain a comparison result; and the comparison result is that the quantity of bits of the code value is less than the quantity of bits of the data, the code value is saved, and at least one of an equivalent comparison, sorting, or a fuzzy search of the data is performed according to the code value. For example, the equivalent comparison of the data may be performed according to the code value, the sorting of the data may be performed according to the code value, or the fuzzy search may be performed on the data according to the code value. The following separately describes the foregoing application operations of the data that are performed according to the code value.

Specifically, in another embodiment, the data is an ID type string, the application operation includes the equivalent comparison, and the performing an application operation of the data according to the code value includes: performing the equivalent comparison of the data according to the code value.

Further, in the another embodiment, the performing the equivalent comparison of the data according to the code value includes: when the code value is equal to a to-be-compared code value, determining that the data and data corresponding to the to-be-compared code value are the same data.

It should be understood that the to-be-compared code value is a code value of to-be-compared data (the data corresponding to the to-be-compared code value). Specifically, performing an equivalent comparison of code values may be understood as performing data matching. For example, when code values of two pieces of data are equal, it may be determined that the two pieces of data corresponding to the two code values are the same data, that is, matching is successful. When the two code values are not equal, it may be determined that the two pieces of data corresponding to the two code values are different data, that is, matching is unsuccessful.

Alternatively, in another embodiment, the data is an ID type string or a letter string of a field, the application operation includes sorting, and the performing an application operation of the data according to the code value includes: performing the sorting of the data according to the code value.

Further, in the another embodiment, performing sorting of the data according to the code value includes: determining a location of the code value in a to-be-sorted code value according to the code value. The location of the code value is used to represent a location of the data in data corresponding to the to-be-sorted code value.

Specifically, for example, the performing sorting of the data may be understood as sorting multiple pieces of data. For example, there are five pieces of data, the five pieces of data are corresponding to five code values, and the five code values are sorted in ascending order. For example, when a current code value is the fourth code value in the five code values, a piece of data corresponding to the fourth code value is sorted in the fourth place in the five pieces of data.

In existing database implementation, a compression technology and a search operation are generally separately considered, that is, a data storage technology and a relevant search optimization technology are independently considered. An important function of a database is to save and record related information such as important description of things of interest and development of the things. Therefore, when the things are tediously described, a large amount of storage space is occupied, and in addition, search inconvenience is brought. When multiple search operations (for example, string comparison and string sorting) are performed on data in the prior art, lower search efficiency is caused because characters in a string need to be gradually compared. However, in this embodiment of the present invention, an arithmetic decoding process is not required, data comparison (matching) is directly performed according to code values corresponding to data and/or data sorting is performed according to the code values, and a search operation of data of a complex type is equivalent to a search of a code value, which is quicker and simpler.

For example, in an application scenario in which only digits or letters occur in data (in this case, probabilities are allocated to the digits or the letters), for example, the data is non-Latin letter text, for example, Chinese, Korean, and Japanese each have corresponding phonetic symbols. Corresponding text may be converted into a phonetic symbol representation manner of the corresponding text, that is, a string including only letters, and then the data is encoded by using an arithmetic coding algorithm, so as to obtain a code value interval; when a code value corresponding to the data exists in the code value interval, it is determined that a quantity of bits of the code value is less than a quantity of bits of the data; and the code value is saved. A field corresponding to the data is sorted according to the code value.

For example, distribution of appearance probabilities of Chinese phonetic letters (all Chinese phonetic letters) is A (0.107), B (0.014), C (0.017), D (0.030), E (0.062), F (0.009), G (0.060), H (0.067), I (0.141), J (0.023), K (0.008), L (0.017), M (0.014), N (0.117), 0 (0.065), P (0.008), Q (0.013), R (0.006), S (0.026), T (0.015), U (0.096), V (0.001), W (0.010), X (0.020), Y (0.028), and Z (0.026). As shown inFIG. 2, fields “excellent” (you xiu in Chinese Pinyin), “good” (liang hao in Chinese Pinyin), and “pass” (ji ge in Chinese Pinyin) are sorted according to a letter sequence and are encoded. A letter string corresponding to “excellent” is “you xiu”, and a code value corresponding to “excellent” is 0.96684845. A letter string corresponding to “good” is “liang hao”, and a code value corresponding to “good” is 0.544375656. A letter string corresponding to “pass” is “ji ge”, and a code value corresponding to “pass” is 0.516228. Sequential sorting of the code values in ascending order is 0.516228, 0.544375656, and 0.96684845, and 0.516228, 0.544375656, and 0.96684845 are corresponding to “pass”, “good”, and “excellent” respectively.

Alternatively, in another embodiment, the application operation includes the fuzzy search, and the performing an application operation of the data according to the code value includes: performing the fuzzy search on the data according to the code value.

Further, in the another embodiment, the performing the fuzzy search on the data according to the code value includes: determining, according to whether the code value falls within an encoding interval of a prefix string that needs to be fuzzily searched, whether the data includes the prefix string. When the code value falls within the encoding interval of the prefix string that needs to be fuzzily searched, the data includes the prefix string, or when the code value falls outside the encoding interval of the prefix string that needs to be fuzzily searched, the data does not include the prefix string.

In other words, when the code value falls within the encoding interval of the prefix string that needs to be fuzzily searched, the data meets the fuzzy search, or when the code value falls outside the encoding interval of the prefix string that needs to be fuzzily searched, the data does not meet the fuzzy search.

Specifically, when a string is compressed by using arithmetic coding, a series of code values is obtained. The code values come from encoding intervals obtained after the string is encoded, and the encoding intervals are not mutually orthogonal. It is further noted that the encoding intervals of the string are always included in encoding intervals of a prefix string of the string. For example, encoding intervals of a string “A12986572” are definitely included in encoding intervals of a prefix string such as “A1298” or “A12”. A fuzzy search may be performed provided that whether the code values fall within an encoding interval of a prefix string that needs to be fuzzily searched is determined. For example, for integer set space {0, 1, 2, 3}, appearance probability distribution is {0.2, 0.5, 0.2, 0.1}. For a fuzzy search %210xxx, the fuzzy search is performed on data 212132, 210312, 210231, and 211123. As shown inFIG. 3, a code value interval of “210” is [0.74, 0.76], and code value intervals corresponding to 212132, 210312, 210231, and 211123 are 0.8238, 0.7592, 0.7576, and 0.7923. Because 0.7592 and 0.7576 fall within an encoding interval [0.74, 0.76] of the prefix string that needs to be fuzzily searched, and 0.8238 and 0.7923 fall outside the encoding interval [0.74, 0.76] of the prefix string that needs to be fuzzily searched, 210312 and 210231 meet the fuzzy search, and 212132 and 211123 do not meet the fuzzy search.

Therefore, in this embodiment of the present invention, in addition, in a fuzzy search operation, when a quantity of index characters is greater than 2, an operation required for determining an interval location may be skipped.

Optionally, in another embodiment, in step110, the data is encoded by using the arithmetic coding algorithm, so as to obtain an encoding interval; the encoding interval of the data is re-extended, so as to obtain a re-extended encoding interval and record a location of a corresponding re-extended character; and the data continues to be encoded according to the re-extended encoding interval by using the arithmetic coding algorithm, so as to obtain the code value interval. The location of the corresponding re-extended character is recorded, so that a decoding device obtains a location of a re-scaled character according to the code value and the location of the re-extended character, performs decoding interval re-scaling according to the location of the re-scaled character, and obtains the data.

Specifically, in this embodiment of the present invention, an encoding interval corresponding to any character may be re-extended, or encoding interval re-extension may be performed according to a preset condition.

Therefore, in this embodiment of the present invention, encoding interval re-extension may be performed on an encoding interval of data. Because the encoding interval is re-extended, a code value interval is accordingly expanded. Therefore, in this embodiment of the present invention, an appropriate code value can be obtained more easily from an expanded code value interval, which avoids incorrect encoding, and implements correct encoding. In addition, in this embodiment of the present invention, performing the interval re-extension can use space with a limited quantity of bits to represent sufficiently long string data.

Further, in the another embodiment, in step110, when at least one of the following conditions is met, the encoding interval of the data is re-extended, so as to obtain the re-extended encoding interval: a quantity of encoded characters of the data reaches a character quantity threshold or a length of the encoding interval of the data is less than an interval threshold.

Specifically, when that a quantity of characters of the data is greater than the preset character quantity threshold is met, in step110, the data is encoded by using the arithmetic coding algorithm, or when the quantity of encoded characters of the data reaches the preset character quantity threshold, an encoding interval of a character of the data corresponding to the preset character quantity threshold is re-extended and the location of the corresponding re-extended character is recorded, and the data continues to be encoded according to the re-extended encoding interval by using the arithmetic coding algorithm, so as to obtain the code value interval.

In other words, when a string of the data is gradually encoded by using the arithmetic coding algorithm, when the quantity of encoded characters of the data reaches the preset character quantity threshold, the encoding interval of the character of the data corresponding to the preset character quantity threshold is re-extended, and the location of the corresponding re-extended character is recorded, and the data continues to be encoded according to the re-extended encoding interval by using the arithmetic coding algorithm, so as to obtain the code value interval.

For example, current data includes a string including 12 characters, and a preset character quantity threshold is 10. When the 12 strings of the current data are encoded by using an arithmetic coding algorithm, an encoding interval of the tenth character in the 12 strings is re-extended, and it is recorded that a location of a re-extended character is a location of the tenth character. Then, the data (the eleventh and the twelfth characters) continues to be encoded according to a re-extended encoding interval by using the arithmetic coding algorithm, and finally a code value interval (an encoding interval corresponding to the twelfth character) is obtained.

With an increase of a length of a string of data, a length of an encoding interval obtained by means of encoding is increasingly shorter, and it is difficult to obtain an appropriate code value from a shorter encoding interval length. Therefore, to avoid, as much as possible, possible occurrence of a problem that arithmetic coding cannot be correctly performed, in this embodiment of the present invention, encoding interval re-extension is performed when a quantity of characters reaches a preset character quantity threshold. Because an encoding interval is re-extended, a code value interval is accordingly expanded. Therefore, in this embodiment of the present invention, an appropriate code value can be obtained more easily from an expanded code value interval, which avoids incorrect encoding, and implements correct encoding. In addition, in this embodiment of the present invention, performing the interval re-extension can use space with a limited quantity of bits to represent sufficiently long string data.

When that the length of the encoding interval of the data is less than the preset threshold is met, in step110, the encoding interval of the data is re-extended; and the data continues to be encoded according to the re-extended encoding interval by using the arithmetic coding algorithm, so as to obtain the code value interval.

In other words, when a string of the data is gradually encoded by using the arithmetic coding algorithm, an encoding interval is increasingly smaller. When the encoding interval is less than the preset threshold, the encoding interval less than the preset threshold is re-extended, and then a to-be-encoded string in the data continues to be encoded according to the re-extended encoding interval by using the arithmetic coding algorithm, and finally, the code value interval is obtained.

For example, current data includes a string including 12 characters, and a preset threshold is 0.05. When the 12 strings of the current data are encoded by using an arithmetic coding algorithm and when an encoding interval is less than 0.05, the encoding interval of the data is re-extended. For example, when an encoding interval of the seventh character is 0.1, and an encoding interval of the eighth character is 0.04, the encoding interval 0.04 of the eighth character is re-extended, for example, 0.04 is extended to 1, 10, or the like. The data (the ninth to twelfth characters) continues to be encoded according to a re-extended encoding interval (1 or 10) by using the arithmetic coding algorithm, so as to obtain a code value interval (an encoding interval corresponding to the twelfth character).

With an increase of a length of a string of data, a length of an encoding interval obtained by means of encoding is increasingly shorter, and it is difficult to obtain an appropriate code value from a shorter encoding interval length. Therefore, to avoid, as much as possible, possible occurrence of a problem that arithmetic coding cannot be correctly performed, in this embodiment of the present invention, encoding interval re-extension is performed. Because an encoding interval is re-extended, a code value interval is accordingly expanded. Therefore, in this embodiment of the present invention, an appropriate code value can be obtained more easily from an expanded code value interval, which avoids incorrect encoding, and implements correct encoding. In addition, in this embodiment of the present invention, performing the interval re-extension can use space with a limited quantity of bits to represent sufficiently long string data.

It should be noted that all information for performing the interval extension is also sent to a decoder, that is, when transferring a binary code value to the decoder, an encoder also sends shift mechanism information to the decoder, so as to synchronize information and ensure that a correct decoding result is obtained during decoding.

FIG. 4is a schematic flowchart of a data processing method according to another embodiment of the present invention. The method inFIG. 4may be executed by a data processing device, and specifically, may be executed by an encoding device. As shown inFIG. 4, the method includes the following steps.

410. Encode data by using an arithmetic coding algorithm, so as to obtain an encoding interval.

420. Re-extend the encoding interval of the data, so as to obtain a re-extended encoding interval.

Specifically, in this embodiment of the present invention, an encoding interval corresponding to any character may be re-extended, or encoding interval re-extension may be performed according to a preset condition.

430. Continue to encode the data according to the re-extended encoding interval by using the arithmetic coding algorithm, so as to obtain a code value interval.

440. Obtain a code value according to the code value interval.

450. Save the code value.

Therefore, in this embodiment of the present invention, encoding interval re-extension is performed on an encoding interval of data. Because the encoding interval is re-extended, a code value interval is accordingly expanded. Therefore, in this embodiment of the present invention, an appropriate code value can be obtained more easily from an expanded code value interval, which avoids incorrect encoding, and implements correct encoding. In addition, in this embodiment of the present invention, performing the interval re-extension can use space with a limited quantity of bits to represent sufficiently long string data.

Further, in another embodiment, in step420, when at least one of the following conditions is met, the encoding interval of the data is re-extended, so as to obtain the re-extended encoding interval and record a location of a corresponding re-extended character: a quantity of encoded characters of the data reaches a character quantity threshold or a length of the encoding interval of the data is less than an interval threshold. The location of the corresponding re-extended character is recorded, so that a decoding device obtains a location of a re-scaled character according to the code value and the location of the re-extended character, performs decoding interval re-scaling according to the location of the re-scaled character, and obtains the data.

Specifically, when that a quantity of characters of the data is greater than the preset character quantity threshold is met, the data is encoded by using the arithmetic coding algorithm, or when the quantity of encoded characters of the data reaches the preset character quantity threshold, an encoding interval of a character of the data corresponding to the preset character quantity threshold is re-extended, and the data continues to be encoded according to the re-extended encoding interval by using the arithmetic coding algorithm, so as to obtain the code value interval.

In other words, when a string of the data is gradually encoded by using the arithmetic coding algorithm, when the quantity of encoded characters of the data reaches the preset character quantity threshold, the encoding interval of the character of the data corresponding to the preset character quantity threshold is re-extended, and the data continues to be encoded according to the re-extended encoding interval by using the arithmetic coding algorithm, so as to obtain the code value interval.

For example, current data includes a string including 12 characters, and a preset character quantity threshold is 10. When the 12 strings of the current data are encoded by using an arithmetic coding algorithm, an encoding interval of the tenth character in the 12 strings is re-extended, and it is recorded that a location of a re-extended character is a location of the tenth character. Then, the data (the eleventh and the twelfth characters) continues to be encoded according to a re-extended encoding interval by using the arithmetic coding algorithm, and finally a code value interval (an encoding interval corresponding to the twelfth character) is obtained.

With an increase of a length of a string of data, a length of an encoding interval obtained by means of encoding is increasingly shorter, and it is difficult to obtain an appropriate code value from a shorter encoding interval length. Therefore, to avoid, as much as possible, possible occurrence of a problem that arithmetic coding cannot be correctly performed, in this embodiment of the present invention, encoding interval re-extension is performed when a quantity of characters reaches a preset character quantity threshold. Because an encoding interval is re-extended, a code value interval is accordingly expanded. Therefore, in this embodiment of the present invention, an appropriate code value can be obtained more easily from an expanded code value interval, which avoids incorrect encoding, and implements correct encoding. In addition, in this embodiment of the present invention, performing the interval re-extension can use space with a limited quantity of bits to represent sufficiently long string data.

When that the length of the encoding interval of the data is less than the preset threshold is met, the encoding interval of the data is re-extended; and the data continues to be encoded according to the re-extended encoding interval by using the arithmetic coding algorithm, so as to obtain the code value interval.

In other words, when a string of the data is gradually encoded by using the arithmetic coding algorithm, an encoding interval is increasingly smaller. When the encoding interval is less than the preset threshold, the encoding interval less than the preset threshold is re-extended, and then a to-be-encoded string in the data continues to be encoded according to the re-extended encoding interval by using the arithmetic coding algorithm, and finally, the code value interval is obtained.

For example, current data includes a string including 12 characters, and a preset threshold is 0.05. When the 12 strings of the current data are encoded by using an arithmetic coding algorithm and when an encoding interval is less than 0.05, the encoding interval of the data is re-extended. For example, when an encoding interval of the seventh character is 0.1, and an encoding interval of the eighth character is 0.04, the encoding interval 0.04 of the eighth character is re-extended, for example, 0.04 is extended to 1, 10, or the like. The data (the ninth to twelfth characters) continues to be encoded according to a re-extended encoding interval (1 or 10) by using the arithmetic coding algorithm, so as to obtain a code value interval (an encoding interval corresponding to the twelfth character).

With an increase of a length of a string of data, a length of an encoding interval obtained by means of encoding is increasingly shorter, and it is difficult to obtain an appropriate code value from a shorter encoding interval length. Therefore, to avoid, as much as possible, possible occurrence of a problem that arithmetic coding cannot be correctly performed, in this embodiment of the present invention, encoding interval re-extension is performed. Because an encoding interval is re-extended, a code value interval is accordingly expanded. Therefore, in this embodiment of the present invention, an appropriate code value can be obtained more easily from an expanded code value interval, which avoids incorrect encoding, and implements correct encoding. In addition, in this embodiment of the present invention, performing the interval re-extension can use space with a limited quantity of bits to represent sufficiently long string data.

It should be noted that all information for performing the interval extension is also sent to a decoder, that is, when transferring a binary code value to the decoder, an encoder also sends shift mechanism information to the decoder, so as to synchronize information and ensure that a correct decoding result is obtained during decoding.

FIG. 5is a schematic flowchart of a data processing method according to another embodiment of the present invention. The method inFIG. 5may be executed by a data processing device, and specifically, may be executed by a decoding device. As shown inFIG. 5, the method includes the following steps.

510. Obtain a code value of data and a location of a re-extended character.

520. Decode the code value of the data by using an arithmetic coding algorithm, so as to obtain a decoding interval.

530. Re-scale the decoding interval of the data according to the location of the re-extended character, so as to obtain a re-scaled decoding interval.

540. Continue to decode the data according to the re-scaled decoding interval by using the arithmetic coding algorithm, so as to obtain the data.

Therefore, in this embodiment of the present invention, decoding interval re-scaling is performed according to a code value in a re-extended encoding interval, which avoids incorrect decoding, and implements correct decoding.

Specifically, in another embodiment, in step530, a location of a re-scaled character is determined according to the location of the re-extended character, where the location of the re-extended character is inverse to the location of the re-scaled character; and the decoding interval of the data is re-scaled according to the location of the re-scaled character, so as to obtain the re-scaled decoding interval: a quantity of encoded characters of the data reaches a character quantity threshold and a length of an encoding interval of the data is less than an interval threshold.

It should be understood that the location of the re-extended character is inverse to the location of the re-scaled character, in other words, the location of the re-extended character is opposite to (or symmetric with) the location of the re-scaled character. For example, current data includes a string including 12 characters, an encoding interval of the tenth character in the 12 strings is re-extended, a location of a re-extended character is a location of the tenth character, and it may be determined that a location of a re-scaled character is a location of the third character according to the location of the tenth characters.

It should be understood that the data processing method inFIG. 5is corresponding to the data processing method inFIG. 4, and a difference lies in that a decoding process inFIG. 5is an inverse operation of an encoding process inFIG. 4. The method inFIG. 5may be obtained by using an inverse process of the process inFIG. 4. To avoid repetition, details are not described herein.

FIG. 6is a schematic flowchart of a data processing method according to another embodiment of the present invention. The method inFIG. 6may be executed by a data processing device. As shown inFIG. 6, the method includes the following steps.

610. Encode data by using an arithmetic coding algorithm, so as to obtain a code value interval.

620. Obtain a code value according to the code value interval.

630. Save the code value.

640. Perform an application operation of the data according to the code value, where the application operation includes at least one of an equivalent comparison, sorting, or a fuzzy search.

Therefore, in this embodiment of the present invention, a code value is obtained by means of data encoding, at least one application of an equivalent comparison, sorting, or a fuzzy search of data is performed according to the code value, which is different from the prior art that an equivalent comparison, sorting, and a fuzzy search of data are performed according to the source data, and an application of the original complex data is equivalent to corresponding processing performed by using the code value, which is quicker and simpler.

Optionally, in another embodiment, the data is an ID type string, the application operation includes the equivalent comparison, and in step640, the equivalent comparison of the data is performed according to the code value.

Further, in the another embodiment, in step640, when the code value is equal to a to-be-compared code value, it is determined that the data and data corresponding to the to-be-compared code value are the same data.

Specifically, performing an equivalent comparison of code values may be understood as performing data matching. For example, when two code values are equal, it may be determined that pieces of data corresponding to the two code values are the same data, that is, matching is successful. When the two code values are not equal, it may be determined that the pieces of data corresponding to the two code values are different data, that is, matching is unsuccessful.

Alternatively, in another embodiment, the data is an ID type string or a letter string of a field, the application operation includes sorting, and in step640, the sorting of the data is performed according to the code value.

Further, in the another embodiment, in step640, a location of the code value in a to-be-sorted code value is determined according to the code value. The location of the code value is used to represent a location of the data in data corresponding to the to-be-sorted code value.

Specifically, for example, performing sorting of the data may be understood as sorting multiple pieces of data. For example, there are five pieces of data, the five pieces of data are corresponding to five code values, and the five code values are sorted in ascending order. For example, when a current code value is the fourth code value in the five code values, a piece of data corresponding to the fourth code value is sorted in the fourth place in the five pieces of data.

In existing database implementation, a compression technology and a search operation are generally separately considered, that is, a data storage technology and a relevant search optimization technology are independently considered. An important function of a database is to save and record related information such as important description of things of interest and development of the things. Therefore, when the things are tediously described, a large amount of storage space is occupied, and in addition, search inconvenience is brought. When multiple search operations (for example, string comparison and string sorting) are performed on data in the prior art, lower search efficiency is caused because characters in a string need to be gradually compared. However, in this embodiment of the present invention, an arithmetic decoding process is not required, data comparison (matching) is directly performed according to code values corresponding to data and/or data sorting is performed according to the code values, and a search operation of data of a complex type is equivalent to a search of a code value, which is quicker and simpler.

For example, in an application scenario in which only digits or letters occur in data (in this case, probabilities are allocated to the digits or the letters), for example, the data is non-Latin letter text, for example, Chinese, Korean, and Japanese each have corresponding phonetic symbols. Corresponding text may be converted into a phonetic symbol representation manner of the corresponding text, that is, a string including only letters, and then the data is encoded by using an arithmetic coding algorithm, so as to obtain a code value interval; when a code value corresponding to the data exists in the code value interval, it is determined that a quantity of bits of the code value is less than a quantity of bits of the data; and the code value is saved. A field corresponding to the data is sorted according to the code value.

For example, distribution of appearance probabilities of Chinese phonetic letters (all Chinese phonetic letters) is A (0.107), B (0.014), C (0.017), D (0.030), E (0.062), F (0.009), G (0.060), H (0.067), I (0.141), J (0.023), K (0.008), L (0.017), M (0.014), N (0.117), O (0.065), P (0.008), Q (0.013), R (0.006), S (0.026), T (0.015), U (0.096), V (0.001), W (0.010), X (0.020), Y (0.028), and Z (0.026). As shown inFIG. 2, fields “excellent”, “good”, and “pass” are sorted according to a letter sequence and are encoded. A letter string corresponding to “excellent” is “you xiu”, and a code value corresponding to “excellent” is 0.96684845. A letter string corresponding to “good” is “liang hao”, and a code value corresponding to “good” is 0.544375656. A letter string corresponding to “pass” is “ji ge”, and a code value corresponding to “pass” is 0.516228. Sequential sorting of the code values in ascending order is 0.516228, 0.544375656, and 0.96684845, and 0.516228, 0.544375656, and 0.96684845 are corresponding to “pass”, “good”, and “excellent” respectively.

Alternatively, in another embodiment, the application operation includes the fuzzy search, and in step640, the fuzzy search is performed on the data according to the code value.

Further, in the another embodiment, in step640, it is determined, according to whether the code value falls within an encoding interval of a prefix string that needs to be fuzzily searched, whether the data includes the prefix string. When the code value falls within the encoding interval of the prefix string that needs to be fuzzily searched, the data includes the prefix string, or when the code value falls outside the encoding interval of the prefix string that needs to be fuzzily searched, the data does not include the prefix string.

In other words, when the code value falls within the encoding interval of the prefix string that needs to be fuzzily searched, the data meets the fuzzy search, or when the code value falls outside the encoding interval of the prefix string that needs to be fuzzily searched, the data does not meet the fuzzy search.

Specifically, when a string is compressed by using arithmetic coding, a series of code values is obtained. The code values come from encoding intervals obtained after the string is encoded, and the encoding intervals are not mutually orthogonal. It is further noted that the encoding intervals of the string are always included in encoding intervals of a prefix string of the string. For example, encoding intervals of a string “A12986572” are definitely included in encoding intervals of a prefix string such as “A1298” or “A12”. A fuzzy search may be performed provided that whether the code values fall within an encoding interval of a prefix string that needs to be fuzzily searched is determined. For example, for integer set space {0, 1, 2, 3}, appearance probability distribution is {0.2, 0.5, 0.2, 0.1}. For a fuzzy search %210xxx, the fuzzy search is performed on data 212132, 210312, 210231, and 211123. As shown inFIG. 3, a code value interval of “210” is [0.74, 0.76], and code value intervals corresponding to 212132, 210312, 210231, and 211123 are 0.8238, 0.7592, 0.7576, and 0.7923. Because 0.7592 and 0.7576 fall within an encoding interval [0.74, 0.76] of the prefix string that needs to be fuzzily searched, and 0.8238 and 0.7923 fall outside the encoding interval [0.74, 0.76] of the prefix string that needs to be fuzzily searched, 210312 and 210231 meet the fuzzy search, and 212132 and 211123 do not meet the fuzzy search.

Therefore, in this embodiment of the present invention, in addition, in a fuzzy search operation, when a quantity of index characters is greater than 2, an operation required for determining an interval location may be skipped.

The foregoing describes the data processing method in the embodiments of the present invention in detail with reference toFIG. 1toFIG. 6. The following describes the embodiments of the present invention in more detail with reference to specific examples inFIG. 7toFIG. 11. It should be noted that the examples inFIG. 7toFIG. 11are merely intended to help a person skilled in the art understand the embodiments of the present invention instead of limiting the embodiments of the present invention to a specific value or a specific scenario shown in the examples. A person skilled in the art certainly can make various equivalent modifications or changes according to the examples provided inFIG. 7toFIG. 11, and such modifications or changes also fall within the scope of the embodiments of the present invention.

FIG. 7is a schematic flowchart of a data processing method according to another embodiment of the present invention. The method inFIG. 7displays a process in which whether arithmetic coding is used is evaluated according to a gain. The method inFIG. 7includes the following steps.

Specifically, the sequence may be a string sequence of source data, for example, may be a data type combining digits and letters, or may be a data type with only digits or only letters, for example, the inputted sequence may be a data document number, an index number of a book, a bus license plate number, a product document number, an IMSI, or an IMEI, or phonetic symbols corresponding to Chinese, Korean, or Japanese, or the like.

Specifically, the encoding interval of the source data is determined according to an arithmetic coding algorithm.

Specifically, it is determined whether a code value corresponding to the data exists in the encoding interval. In other words, it is determined whether a code value may be selected. If a code value may be selected, step750is performed. If no code value may be selected, step790is performed.

Specifically, when a code value corresponding to the data exists in a code value interval, it is determined whether a quantity of bits of the code value meets a requirement, for example, it is determined whether the quantity of bits of the code value is less than a quantity of bits of the data.

760. Is a requirement met?

When the requirement is met, step770is performed. Otherwise, when the requirement is not met, step780is performed.

Specifically, operations such as an equivalent comparison, sorting, and a fuzzy search may be performed according to the code value.

Specifically, in this embodiment of the present invention, it may be determined whether an appropriate code value can be obtained from encoding interval obtained last time. If no appropriate code value can be obtained, the data is directly saved without using the arithmetic coding. When an appropriate code value may be obtained, it is determined whether a required quantity of bits used to represent the code value is greater than a required quantity of bits used to represent the source data. If there is no compression gain, use of the arithmetic coding is abandoned.

Therefore, in this embodiment of the present invention, data is encoded by using an arithmetic coding algorithm, so as to obtain a code value interval; when a code value corresponding to the data exists in the code value interval, it is determined that a quantity of bits of the code value is less than a quantity of bits of the data, and the code value is saved. In this embodiment of the present invention, a gain is determined, which can reduce incorrect encoding, and reduce data storage space.

FIG. 8is a schematic flowchart of a data processing method according to another embodiment of the present invention. The method inFIG. 8displays an arithmetic coding process and an arithmetic decoding process that are based on encoding interval re-extension. The method inFIG. 8includes the following steps.

810. Input a source sequence.

Specifically, the sequence may be a string sequence of source data, for example, may be a data type combining digits and letters, or may be a data type with only digits or only letters, for example, the inputted sequence may be a data document number, an index number of a book, a bus license plate number, a product document number, an IMSI, or an IMEI, or phonetic symbols corresponding to Chinese, Korean, or Japanese, or the like.

Specifically, the source model includes a probability value of each character. A data processing device may perform arithmetic coding on the source sequence according to the source model.

Specifically, a string of the data is gradually encoded according to the source model by using an arithmetic coding algorithm.

840. Is a preset condition met?

Specifically, it is determined whether a quantity of encoded characters of the data reaches a preset character quantity threshold or whether a length of an encoding interval of the data is less than a preset threshold. If the quantity of encoded characters of the data reaches the preset character quantity threshold or the length of the encoding interval of the data is less than the preset threshold, step850is performed. If the quantity of encoded characters of the data does not reach the preset character quantity threshold or the length of the encoding interval of the data is not less than the preset threshold, step840is performed.

When the preset condition is met, the encoding interval of the data is re-extended; the data continues to be encoded according to a re-extended encoding interval by using the arithmetic coding algorithm, so as to obtain a code value interval; and finally, a code value of the source sequence is obtained.

For example, current data includes a string including 12 characters, and a preset threshold is 0.05. When the 12 strings of the current data are encoded by using an arithmetic coding algorithm and when an encoding interval is less than 0.05, the encoding interval of the data is re-extended. For example, when an encoding interval of the seventh character is 0.1, and an encoding interval of the eighth character is 0.04, the encoding interval 0.04 of the eighth character is re-extended, for example, 0.04 is extended to 1, 10, or the like. The data (the ninth to twelfth characters) continues to be encoded according to a re-extended encoding interval (1 or 10) by using the arithmetic coding algorithm, so as to obtain a code value interval (an encoding interval corresponding to the twelfth character).

For another example, current data includes a string including 12 characters, and a preset threshold is 0.05. When the 12 strings of the current data are encoded by using an arithmetic coding algorithm and when an encoding interval is less than 0.05, the encoding interval of the data is re-extended. For example, when an encoding interval of the seventh character is 0.1, and an encoding interval of the eighth character is 0.04, the encoding interval 0.04 of the eighth character is re-extended, for example, 0.04 is extended to 1, 10, or the like. The data (the ninth to twelfth characters) continues to be encoded according to a re-extended encoding interval (1 or 10) by using the arithmetic coding algorithm, so as to obtain a code value interval (an encoding interval corresponding to the twelfth character).

Specifically, for example, an inverse operation is accordingly performed, that is, interval scaling is performed according to the source model and encoding information during decoding if interval extension is performed during encoding.

It should be noted that all information for performing the interval extension is also sent to a decoder, that is, when transferring a binary code value to the decoder, an encoder also sends shift mechanism information to the decoder, so as to synchronize information and ensure that a correct decoding result is obtained during decoding.

Corresponding to the source model in step820, specifically, the source model includes a probability value of each character. A data decoding device may perform arithmetic decoding on the source sequence according to the source model.

880. Obtain a decoded sequence.

Specifically, the decoded sequence may be the same as the source sequence.

With an increase of a length of a string of data, a length of an encoding interval obtained by means of encoding is increasingly shorter, and it is difficult to obtain an appropriate code value from a shorter encoding interval length. Therefore, possible occurrence of a problem that arithmetic coding cannot be correctly performed is avoided as much as possible, and an appropriate code value can be obtained more easily, which avoids incorrect encoding and implements correct encoding. In addition, in this embodiment of the present invention, performing the interval re-extension can use space with a limited quantity of bits to represent sufficiently long string data.

FIG. 9is a schematic flowchart of a data processing method according to another embodiment of the present invention. The method inFIG. 9displays an arithmetic coding process and a search operation process of an ID type string. The method inFIG. 9includes the following steps.

910. Obtain an ID type string.

Specifically, the ID type string may be a string with a lower appearance probability of a letter and a higher appearance probability of a digit, such as a data document number, an index number of a book, or a bus license plate number. For example, generally, a letter occupies only one bit or two bits in a string.

920. Design a probabilistic model.

Specifically, the probabilistic model includes a probability of each character.

Specifically, the string of data is gradually encoded according to the probabilistic model by using an arithmetic coding algorithm.

940. Evaluation and determining.

Specifically, it is determined whether a code value corresponding to the data exists in an encoding interval. In other words, it is determined whether a code value may be selected. If a code value may be selected, it is determined whether a quantity of bits of the code value meets a requirement, for example, it is determined whether the quantity of bits of the code value is less than a quantity of bits of the data.

950. Is a requirement met?

If the requirement is met, step970is performed. If the requirement is not met, step960is performed.

970. Obtain a code value.

Specifically, the code value corresponding to the ID type string is determined from the code value interval.

Specifically, an equivalent comparison operation of the data may be performed according to the code value. When the code value is equal to a to-be-compared code value, it is determined that the data and data corresponding to the to-be-compared code value are the same data. Specifically, performing an equivalent comparison of code values may be understood as performing data matching. For example, when two code values are equal, it may be determined that pieces of data corresponding to the two code values are the same data, that is, matching is successful. When the two code values are not equal, it may be determined that the pieces of data corresponding to the two code values are different data, that is, matching is unsuccessful.

Sorting of the data may further be performed according to the code value. For example, a location of the code value in a to-be-sorted code value is determined according to the code value, where the location of the code value is used to represent a location of the data in data corresponding to the to-be-sorted code value. Specifically, performing the sorting of the data may be understood as sorting multiple pieces of data. For example, there are five pieces of data, the five pieces of data are corresponding to five code values, and the five code values are sorted in ascending order. For example, when a current code value is the fourth code value in the five code values, a piece of data corresponding to the fourth code value is sorted in the fourth place in the five pieces of data.

FIG. 10is a schematic flowchart of a data processing method according to another embodiment of the present invention. The method inFIG. 10displays an arithmetic coding process and a field sorting process of a letter string of a field. The method inFIG. 10includes the following steps.

Specifically, the field may be a Chinese field, a Korean field, or a Japanese field, and is not limited thereto in this embodiment of the present invention. The field may further be another field converted into a letter string by using phonetic symbols. For example, the field may be Chinese fields such as “excellent”, “good”, and “pass”.

Specifically, the field is converted into the letter string. For example, letter strings corresponding to “excellent”, “good”, and “pass” are “you xiu”, “liang hao”, and “ji ge” respectively.

Specifically, a probability of each letter is obtained. For example, distribution of appearance probabilities of Chinese phonetic letters (all Chinese phonetic letters) is A (0.107), B (0.014), C (0.017), D (0.030), E (0.062), F (0.009), G (0.060), H (0.067), I (0.141), J (0.023), K (0.008), L (0.017), M (0.014), N (0.117), 0 (0.065), P (0.008), Q (0.013), R (0.006), S (0.026), T (0.015), U (0.096), V (0.001), W (0.001), X (0.020), Y (0.028), and Z (0.026).

Specifically, encoding is performed according to the foregoing probabilities of the phonetic letters by using an arithmetic coding algorithm.

1050. Evaluation and determining.

Specifically, it is determined whether a code value corresponding to the data exists in an encoding interval. In other words, it is determined whether a code value may be selected. If a code value may be selected, it is determined whether a quantity of bits of the code value meets a requirement, for example, it is determined whether the quantity of bits of the code value is less than a quantity of bits of the data.

1060. Is a requirement met?

If the requirement is met, step108ois performed. If the requirement is not met, step1070is performed.

1080. Obtain a code value.

Specifically, the code value corresponding to the letter string is determined from the code value interval.

For example, fields “excellent”, “good”, and “pass” are sorted according to a letter sequence and are encoded. A letter string corresponding to “excellent” is “you xiu”, and a code value corresponding to “excellent” is 0.96684845. A letter string corresponding to “good” is “liang hao”, and a code value corresponding to “good” is 0.544375656. A letter string corresponding to “pass” is “ji ge”, and a code value corresponding to “pass” is 0.516228. Sequential sorting of the code values in ascending order is 0.516228, 0.544375656, and 0.96684845, and 0.516228, 0.544375656, and 0.96684845 are corresponding to “pass”, “good”, and “excellent” respectively.

FIG. 11is a schematic flowchart of a data processing method according to another embodiment of the present invention. The method inFIG. 11displays a fuzzy search process based on arithmetic coding. The method inFIG. 11includes the following steps.

1110. Select an index segment.

Specifically, the prefix string segment that needs to be fuzzily searched is obtained. For example, the index segment is “210”.

Specifically, the index segment is encoded according to an arithmetic coding algorithm.

Specifically, the code value interval of the index segment is obtained. For example, for integer set space {0, 1, 2, 3}, appearance probability distribution is {0.2, 0.5, 0.2, 0.1}. For “210”, a code value interval of “210” is [0.74, 0.76].

1140. Obtain a code value corresponding to a sequence.

Specifically, the code value corresponding to the sequence that needs to be fuzzily searched is obtained. For example, code value intervals corresponding to 212132, 210312, 210231, and 211123 are 0.8238, 0.7592, 0.7576, and 0.7923 respectively.

1150. Check and record.

Specifically, a fuzzy search is performed according to the code value corresponding to the sequence and the code value interval of the index segment and a result is recorded. For example, a code value interval of “210” is [0.74, 0.76], and code value intervals corresponding to 212132, 210312, 210231, and 211123 are 0.8238, 0.7592, 0.7576, and 0.7923 respectively. Because 0.7592 and 0.7576 fall within an encoding interval [0.74, 0.76] of the prefix string that needs to be fuzzily searched, and 0.8238 and 0.7923 fall outside the encoding interval of the prefix string that needs to be fuzzily searched, 210312 and 210231 meet a fuzzy search condition, and 212132 and 211123 do not meet the fuzzy search condition.

1160. Is the process ended?

Specifically, if the fuzzy search ends, step1170is performed. If the fuzzy search does not end, step1140is performed, so as to obtain a code value corresponding to another sequence.

1170. Output a result.

It should be noted that the examples inFIG. 7toFIG. 11are intended to help a person skilled in the art better understand the embodiments of the present invention instead of limiting the scope of the embodiments of the present invention. A person skilled in the art certainly can make various equivalent modifications or changes according to the examples provided inFIG. 7toFIG. 11, and such modifications or changes also fall within the scope of the embodiments of the present invention.

The foregoing describes the data processing method in the embodiments of the present invention in detail with reference toFIG. 1toFIG. 11. The following describes a data processing device in the embodiments of the present invention with reference toFIG. 12toFIG. 19.

FIG. 12is a schematic block diagram of a data processing device according to an embodiment of the present invention. A data processing device1200inFIG. 12may be an encoding device. The data processing device1200inFIG. 12includes an encoding unit1210, an obtaining unit1220, a comparison unit1230, and a first storage unit1240.

Specifically, the encoding unit1210is configured to encode data by using an arithmetic coding algorithm, so as to obtain a code value interval; the obtaining unit1220is configured to obtain, when a code value corresponding to the data exists in the code value interval, the code value according to the code value interval; the comparison unit1230is configured to compare a quantity of bits of the code value with a quantity of bits of the data, so as to obtain a comparison result; and the first storage unit1240is configured to perform a storage operation according to the comparison result.

Therefore, in this embodiment of the present invention, data is encoded by using an arithmetic coding algorithm, so as to obtain a code value interval; when a code value corresponding to the data exists in the code value interval, the code value is obtained according to the code value interval; a quantity of bits of the code value is compared with a quantity of bits of the data, so as to obtain a comparison result; and a storage operation is performed according to the comparison result. In this embodiment of the present invention, incorrect encoding can be reduced, and data storage space can be reduced.

Optionally, in another embodiment, the comparison result is that the quantity of bits of the code value is less than the quantity of bits of the data, and the first storage unit saves the code value according to the comparison result.

Alternatively, in another embodiment, the comparison result is that the quantity of bits of the code value is greater than or equal to the quantity of bits of the data, and the first storage unit saves the data according to the comparison result.

Optionally, in another embodiment, the device further includes an application unit, configured to perform an application operation of the data according to the code value. The application operation includes at least one of an equivalent comparison, sorting, or a fuzzy search.

Optionally, in another embodiment, the data is an ID type string, the application operation includes the equivalent comparison, and the application unit performs the equivalent comparison of the data according to the code value.

Specifically, in the another embodiment, when the code value is equal to a to-be-compared code value, the application unit determines that the data and data corresponding to the to-be-compared code value are the same data.

Alternatively, in another embodiment, the data is an ID type string or a letter string of a field, the application operation includes sorting, and the application unit performs the sorting of the data according to the code value.

Specifically, in the another embodiment, the application unit determines a location of the code value in a to-be-sorted code value according to the code value. The location of the code value is used to represent a location of the data in data corresponding to the to-be-sorted code value.

Alternatively, in another embodiment, the application operation includes the fuzzy search, and the application unit performs the fuzzy search on the data according to the code value.

Specifically, in the another embodiment, the application unit determines, according to whether the code value falls within an encoding interval of a prefix string that needs to be fuzzily searched, whether the data includes the prefix string. When the code value falls within the encoding interval of the prefix string that needs to be fuzzily searched, the data includes the prefix string, or when the code value falls outside the encoding interval of the prefix string that needs to be fuzzily searched, the data does not include the prefix string.

Optionally, in another embodiment, the encoding unit1210encodes the data by using the arithmetic coding algorithm, so as to obtain an encoding interval; re-extends the encoding interval of the data, so as to obtain a re-extended encoding interval; and continues to encode the data according to the re-extended encoding interval by using the arithmetic coding algorithm, so as to obtain the code value interval.

Further, when at least one of the following conditions is met, the encoding unit1210re-extends the encoding interval of the data, so as to obtain the re-extended encoding interval and record a location of a corresponding re-extended character: a quantity of encoded characters of the data reaches a character quantity threshold or a length of the encoding interval of the data is less than an interval threshold.

Optionally, in another embodiment, the device further includes a determining unit, configured to determine, before the obtaining unit1220obtains the code value, whether an appropriate code value corresponding to the data exists in the code value interval.

Optionally, in another embodiment, the device further includes a second storage unit, configured to save the data when no appropriate code value corresponding to the data exists in the code value interval.

It should be understood that the data processing device inFIG. 12is corresponding to the data processing method inFIG. 1, the data processing device inFIG. 12can implement processes of the data processing method inFIG. 1, and for a function of the data processing device inFIG. 12, refer to relevant description of the data processing method inFIG. 1. To avoid repetition, details are not described herein again.

FIG. 13is a schematic block diagram of a data processing device according to another embodiment of the present invention. The data processing device1300inFIG. 13may be an encoding device. The device1300inFIG. 13includes a first encoding unit1310, an extension unit1320, a second encoding unit1330, an obtaining unit1340, and a storage unit1350.

Specifically, the first encoding unit1310is configured to encode data by using an arithmetic coding algorithm, so as to obtain an encoding interval, the extension unit1320is configured to re-extend the encoding interval of the data, so as to obtain a re-extended encoding interval, the second encoding unit1330is configured to continue to encode the data according to the re-extended encoding interval by using the arithmetic coding algorithm, so as to obtain a code value interval, the obtaining unit1340is configured to obtain a code value according to the code value interval, and the storage unit1350is configured to save the code value.

Therefore, in this embodiment of the present invention, encoding interval re-extension is performed on an encoding interval of data. Because the encoding interval is re-extended, a code value interval is accordingly expanded. Therefore, in this embodiment of the present invention, an appropriate code value can be obtained more easily from an expanded code value interval, which avoids incorrect encoding, and implements correct encoding. In addition, in this embodiment of the present invention, performing the interval re-extension can use space with a limited quantity of bits to represent sufficiently long string data.

Optionally, in another embodiment, when at least one of the following conditions is met, the extension unit1320re-extends the encoding interval of the data, so as to obtain the re-extended encoding interval and record a location of a corresponding re-extended character: a quantity of encoded characters of the data reaches a character quantity threshold or a length of the encoding interval of the data is less than an interval threshold.

It should be understood that the data processing device inFIG. 13is corresponding to the data processing method inFIG. 4, the data processing device inFIG. 13can implement processes of the data processing method inFIG. 4, and for a function of the data processing device inFIG. 13, refer to relevant description of the data processing method inFIG. 4. To avoid repetition, details are not described herein again.

FIG. 14is a schematic block diagram of a data processing device according to another embodiment of the present invention. The data processing device1400inFIG. 14may be a decoding device. The device1400inFIG. 14includes a first obtaining unit1410, a first decoding unit1420, a scaling unit1430, and a second decoding unit1440.

Specifically, the first obtaining unit1410is configured to obtain a code value of data and a location of a re-extended character, the first decoding unit1420is configured to decode the code value of the data by using an arithmetic coding algorithm, so as to obtain a decoding interval, the scaling unit1430is configured to re-scale the decoding interval of the data according to the location of the re-extended character, so as to obtain a re-scaled decoding interval, and the second decoding unit1440is configured to continue to decode the data according to the re-scaled decoding interval by using the arithmetic coding algorithm, so as to obtain the data.

Therefore, in this embodiment of the present invention, decoding interval re-scaling is performed according to a code value in a re-extended encoding interval, which avoids incorrect decoding, and implements correct decoding.

Optionally, in another embodiment, the scaling unit1430determines a location of a re-scaled character according to the location of the re-extended character, where the location of the re-extended character is inverse to the location of the re-scaled character; and re-scales the decoding interval of the data according to the location of the re-scaled character, so as o obtain the re-scaled decoding interval.

It should be understood that the data processing device inFIG. 14is corresponding to the data processing method inFIG. 5, the data processing device inFIG. 14can implement processes of the data processing method inFIG. 5, and for a function of the data processing device inFIG. 14, refer to relevant description of the data processing method inFIG. 5. To avoid repetition, details are not described herein again.

FIG. 15is a schematic block diagram of a data processing device according to another embodiment of the present invention. The data processing device1500inFIG. 15may be an encoding device. The device1500inFIG. 15includes an encoding unit1510, an obtaining unit1520, a storage unit1530, and an application unit1540.

Specifically, the encoding unit1510is configured to encode data by using an arithmetic coding algorithm, so as to obtain a code value interval; the obtaining unit1520is configured to obtain a code value according to the code value interval; the storage unit1530is configured to save the code value; and the application unit1540is configured to perform an application operation of the data according to the code value, where the application operation includes at least one of an equivalent comparison, sorting, or a fuzzy search.

Therefore, in this embodiment of the present invention, a code value is obtained by means of data encoding, at least one application of an equivalent comparison, sorting, or a fuzzy search of data is performed according to the code value, which is different from the prior art that an equivalent comparison, sorting, and a fuzzy search of data are performed according to the source data, and an application of the original complex data is equivalent to corresponding processing performed by using the code value, which is quicker and simpler.

Optionally, in another embodiment, the data is an ID type string, the application operation includes the equivalent comparison, and the application unit1540performs the equivalent comparison of the data according to the code value.

Specifically, in the another embodiment, the application unit1540determines, when the code value is equal to a to-be-compared code value, that the data and data corresponding to the to-be-compared code value are the same data.

Alternatively, in another embodiment, the data is an ID type string or a letter string of a field, the application operation includes sorting, and the application unit1540performs the sorting of the data according to the code value.

Specifically, in the another embodiment, the application unit1540determines a location of the code value in a to-be-sorted code value according to the code value. The location of the code value is used to represent a location of the data in data corresponding to the to-be-sorted code value.

Alternatively, in another embodiment, the application operation includes the fuzzy search, and the application unit1540performs the fuzzy search on the data according to the code value.

Specifically, in the another embodiment, the application unit1540determines, according to whether the code value falls within an encoding interval of a prefix string that needs to be fuzzily searched, whether the data includes the prefix string. When the code value falls within the encoding interval of the prefix string that needs to be fuzzily searched, the data includes the prefix string, or when the code value falls outside the encoding interval of the prefix string that needs to be fuzzily searched, the data does not include the prefix string.

It should be understood that the data processing device inFIG. 15is corresponding to the data processing method inFIG. 6, the data processing device inFIG. 15can implement processes of the data processing method inFIG. 6, and for a function of the data processing device inFIG. 15, refer to relevant description of the data processing method inFIG. 6. To avoid repetition, details are not described herein again.

FIG. 16is a schematic block diagram of a data processing device according to another embodiment of the present invention. The data processing device1600inFIG. 16may be an encoding device. The data processing device1600inFIG. 16includes a processor1610, a memory1620, and a bus system1630.

Specifically, the processor1610invokes, by using the bus system1630, code saved in the memory1620and encodes data by using an arithmetic coding algorithm, so as to obtain a code value interval; when a code value corresponding to the data exists in the code value interval, obtains the code value according to the code value interval; compares a quantity of bits of the code value with a quantity of bits of the data, so as to obtain a comparison result; and performs a storage operation according to the comparison result.

In this embodiment of the present invention, data is encoded by using an arithmetic coding algorithm, so as to obtain a code value interval; when a code value corresponding to the data exists in the code value interval, the code value is obtained according to the code value interval; a quantity of bits of the code value is compared with a quantity of bits of the data, so as to obtain a comparison result; and a storage operation is performed according to the comparison result. In this embodiment of the present invention, incorrect encoding can be reduced, and data storage space can be reduced.

The methods disclosed in the foregoing embodiments of the present invention may be applied to the processor1610, or implemented by the processor1610. The processor1610may be an integrated circuit chip and has a signal processing capability. In an implementation process, the steps in the foregoing methods may be completed by using an integrated logic circuit of hardware in the processor1610or an instruction in a form of software. The foregoing processor1610may be a general purpose processor, a digital signal processor (Digital Signal Processor in English, DSP for short), an application-specific integrated circuit (Application Specific Integrated Circuit in English, ASIC for short), a field programmable gate array (Field Programmable Gate Array in English, FPGA for short), or another programmable logic device, a discrete gate, or a transistor logic device, or a discrete hardware assembly, which may implement or execute the methods, steps, and logical block diagrams that are disclosed in the embodiments of the present invention. The general purpose processor may be a microprocessor or the processor may be any conventional processor, or the like. The steps of the methods disclosed with reference to the embodiments of the present invention may be directly executed and completed by a hardware decoding processor, or executed and completed by a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory (Random Access Memory in English, RAM for short), a flash memory, a read-only memory (Read-Only Memory in English, ROM for short), a programmable read-only memory, or an electrically erasable programmable memory, or a register. The storage medium is located in the memory1620. The processor1610reads information in the memory1620, and completes the steps of the foregoing methods with reference to hardware of the processor1610. In addition to a data bus, the bus system1630may further include a power bus, a control bus, a state signal bus, and the like. However, for clarity of description, various buses are marked as the bus system1630in the figure.

Optionally, in another embodiment, the comparison result is that the quantity of bits of the code value is less than the quantity of bits of the data, and the processor1610saves the code value according to the comparison result.

Alternatively, in another embodiment, the comparison result is that the quantity of bits of the code value is greater than or equal to the quantity of bits of the data, and the processor1610saves the data according to the comparison result.

Optionally, in another embodiment, the device further includes: the processor1610is configured to perform an application operation of the data according to the code value. The application operation includes at least one of an equivalent comparison, sorting, or a fuzzy search.

Optionally, in another embodiment, the data is an ID type string, the application operation includes the equivalent comparison, and the processor1610performs the equivalent comparison of the data according to the code value.

Specifically, in the another embodiment, the processor1610determines, when the code value is equal to a to-be-compared code value, that the data and data corresponding to the to-be-compared code value are the same data.

Alternatively, in another embodiment, the data is an ID type string or a letter string of a field, the application operation includes sorting, and the processor1610performs the sorting of the data according to the code value.

Specifically, in the another embodiment, the processor1610determines a location of the code value in a to-be-sorted code value according to the code value. The location of the code value is used to represent a location of the data in data corresponding to the to-be-sorted code value.

Alternatively, in another embodiment, the application operation includes the fuzzy search, and the processor1610performs the fuzzy search on the data according to the code value.

Specifically, in the another embodiment, the processor1610determines, according to whether the code value falls within an encoding interval of a prefix string that needs to be fuzzily searched, whether the data includes the prefix string. When the code value falls within the encoding interval of the prefix string that needs to be fuzzily searched, the data includes the prefix string, or when the code value falls outside the encoding interval of the prefix string that needs to be fuzzily searched, the data does not include the prefix string.

Optionally, in another embodiment, the processor1610encodes the data by using the arithmetic coding algorithm, so as to obtain an encoding interval; re-extends the encoding interval of the data, so as to obtain a re-extended encoding interval; and continues to encode the data according to the re-extended encoding interval by using the arithmetic coding algorithm, so as to obtain the code value interval.

Further, when at least one of the following conditions is met, the processor161ore-extends the encoding interval of the data, so as to obtain the re-extended encoding interval and record a location of a corresponding re-extended character: a quantity of encoded characters of the data reaches a character quantity threshold or a length of the encoding interval of the data is less than an interval threshold.

Optionally, in another embodiment, before obtaining the code value, the processor1610determines whether an appropriate code value corresponding to the data exists in the code value interval.

Optionally, in another embodiment, when no appropriate code value corresponding to the data exists in the code value interval, the processor1610saves the data.

It should be understood that the data processing device inFIG. 16is corresponding to the data processing method inFIG. 1, the data processing device inFIG. 16can implement processes of the data processing method inFIG. 1, and for a function of the data processing device inFIG. 16, refer to relevant description of the data processing method inFIG. 1. To avoid repetition, details are not described herein again.

FIG. 17is a schematic block diagram of a data processing device according to another embodiment of the present invention. The data processing device1700inFIG. 17may be an encoding device. The device1700inFIG. 17includes a processor1710, a memory1720, and a bus system1730.

Specifically, the processor1710invokes, by using the bus system1730, code saved in the memory1720and encodes data by using an arithmetic coding algorithm, so as to obtain an encoding interval; re-extends the encoding interval of the data, so as to obtain a re-extended encoding interval; continues to encode the data according to the re-extended encoding interval by using the arithmetic coding algorithm, so as to obtain a code value interval; obtains a code value according to the code value interval; and saves the code value.

Therefore, in this embodiment of the present invention, encoding interval re-extension is performed on an encoding interval of data. Because the encoding interval is re-extended, a code value interval is accordingly expanded. Therefore, in this embodiment of the present invention, an appropriate code value can be obtained more easily from an expanded code value interval, which avoids incorrect encoding, and implements correct encoding. In addition, in this embodiment of the present invention, performing the interval re-extension can use space with a limited quantity of bits to represent sufficiently long string data.

The methods disclosed in the foregoing embodiments of the present invention may be applied to the processor1710, or implemented by the processor1710. The processor1710may be an integrated circuit chip and has a signal processing capability. In an implementation process, the steps in the foregoing methods may be completed by using an integrated logic circuit of hardware in the processor1710or an instruction in a form of software. The foregoing processor1710may be a general purpose processor, a digital signal processor (Digital Signal Processor in English, DSP for short), an application-specific integrated circuit (Application Specific Integrated Circuit in English, ASIC for short), a field programmable gate array (Field Programmable Gate Array in English, FPGA for short), or another programmable logic device, a discrete gate, or a transistor logic device, or a discrete hardware assembly, which may implement or execute the methods, steps, and logical block diagrams that are disclosed in the embodiments of the present invention. The general purpose processor may be a microprocessor or the processor may be any conventional processor, or the like. The steps of the methods disclosed with reference to the embodiments of the present invention may be directly executed and completed by a hardware decoding processor, or executed and completed by a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory (Random Access Memory in English, RAM for short), a flash memory, a read-only memory (Read-Only Memory in English, ROM for short), a programmable read-only memory, or an electrically erasable programmable memory, or a register. The storage medium is located in the memory1720. The processor1710reads information in the memory1720, and completes the steps of the foregoing methods with reference to hardware of the processor1710. In addition to a data bus, the bus system1730may further include a power bus, a control bus, a state signal bus, and the like. However, for clarity of description, various buses are marked as the bus system1730in the figure.

Optionally, In another embodiment, when at least one of the following conditions is met, the processor1710re-extends the encoding interval of the data, so as to obtain the re-extended encoding interval and record a location of a corresponding re-extended character: a quantity of encoded characters of the data reaches a character quantity threshold or a length of the encoding interval of the data is less than an interval threshold.

It should be understood that the data processing device inFIG. 17is corresponding to the data processing method inFIG. 4, the data processing device inFIG. 17can implement processes of the data processing method inFIG. 4, and for a function of the data processing device inFIG. 17, refer to relevant description of the data processing method inFIG. 4. To avoid repetition, details are not described herein again.

FIG. 18is a schematic block diagram of a data processing device according to another embodiment of the present invention. The data processing device1800inFIG. 18may be a decoding device. The device1800inFIG. 18includes a processor1810, a memory1820, and a bus system1830.

Specifically, the processor1810invokes, by using the bus system1830, code saved in the memory1820to obtain a code value of data and a location of a re-extended character; decodes the code value of the data by using an arithmetic coding algorithm, so as to obtain a decoding interval; re-scales the decoding interval of the data according to the location of the re-extended character, so as to obtain a re-scaled decoding interval; and continues to decode the data according to the re-scaled decoding interval by using the arithmetic coding algorithm, so as to obtain the data.

Therefore, in this embodiment of the present invention, decoding interval re-scaling is performed according to a code value in a re-extended encoding interval, which avoids incorrect decoding, and implements correct decoding.

The methods disclosed in the foregoing embodiments of the present invention may be applied to the processor1810, or implemented by the processor1810. The processor1810may be an integrated circuit chip and has a signal processing capability. In an implementation process, the steps in the foregoing methods may be completed by using an integrated logic circuit of hardware in the processor1810or an instruction in a form of software. The foregoing processor181omay be a general purpose processor, a digital signal processor (Digital Signal Processor in English, DSP for short), an application-specific integrated circuit (Application Specific Integrated Circuit in English, ASIC for short), a field programmable gate array (Field Programmable Gate Array in English, FPGA for short), or another programmable logic device, a discrete gate, or a transistor logic device, or a discrete hardware assembly, which may implement or execute the methods, steps, and logical block diagrams that are disclosed in the embodiments of the present invention. The general purpose processor may be a microprocessor or the processor may be any conventional processor, or the like. The steps of the methods disclosed with reference to the embodiments of the present invention may be directly executed and completed by a hardware decoding processor, or executed and completed by a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory (Random Access Memory in English, RAM for short), a flash memory, a read-only memory (Read-Only Memory in English, ROM for short), a programmable read-only memory, or an electrically erasable programmable memory, or a register. The storage medium is located in the memory1820. The processor1810reads information in the memory1820, and completes the steps of the foregoing methods with reference to hardware of the processor1810. In addition to a data bus, the bus system1830may further include a power bus, a control bus, a state signal bus, and the like. However, for clarity of description, various buses are marked as the bus system1830in the figure.

Optionally, in another embodiment, the processor1810determines a location of a re-scaled character according to the location of the re-extended character, where the location of the re-extended character is inverse to the location of the re-scaled character; and re-scales the decoding interval of the data according to the location of the re-scaled character, so as o obtain the re-scaled decoding interval.

It should be understood that the data processing device inFIG. 18is corresponding to the data processing method inFIG. 5, the data processing device inFIG. 18can implement processes of the data processing method inFIG. 5, and for a function of the data processing device inFIG. 18, refer to relevant description of the data processing method inFIG. 5. To avoid repetition, details are not described herein again.

FIG. 19is a schematic block diagram of a data processing device according to another embodiment of the present invention. The data processing device1900inFIG. 19may be an encoding device. The device1900inFIG. 19includes a processor1910, a memory1920, and a bus system1930.

Specifically, the processor1910invokes, by using the bus system1930, code saved in the memory1920and encodes data by using an arithmetic coding algorithm, so as to obtain a code value interval; obtains a code value according to the code value interval; saves the code value; and performs an application operation of the data according to the code value, where the application operation includes at least one of an equivalent comparison, sorting, or a fuzzy search.

Therefore, in this embodiment of the present invention, a code value is obtained by means of data encoding, at least one application of an equivalent comparison, sorting, or a fuzzy search of data is performed according to the code value, which is different from the prior art that an equivalent comparison, sorting, and a fuzzy search of data are performed according to the source data, and an application of the original complex data is equivalent to corresponding processing performed by using the code value, which is quicker and simpler.

The methods disclosed in the foregoing embodiments of the present invention may be applied to the processor1910, or implemented by the processor1910. The processor1910may be an integrated circuit chip and has a signal processing capability. In an implementation process, the steps in the foregoing methods may be completed by using an integrated logic circuit of hardware in the processor1910or an instruction in a form of software. The foregoing processor1910may be a general purpose processor, a digital signal processor (Digital Signal Processor in English, DSP for short), an application-specific integrated circuit (Application Specific Integrated Circuit in English, ASIC for short), a field programmable gate array (Field Programmable Gate Array in English, FPGA for short), or another programmable logic device, a discrete gate, or a transistor logic device, or a discrete hardware assembly, which may implement or execute the methods, steps, and logical block diagrams that are disclosed in the embodiments of the present invention. The general purpose processor may be a microprocessor or the processor may be any conventional processor, or the like. The steps of the methods disclosed with reference to the embodiments of the present invention may be directly executed and completed by a hardware decoding processor, or executed and completed by a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory (Random Access Memory in English, RAM for short), a flash memory, a read-only memory (Read-Only Memory in English, ROM for short), a programmable read-only memory, or an electrically erasable programmable memory, or a register. The storage medium is located in the memory1920. The processor1910reads information in the memory1920, and completes the steps of the foregoing methods with reference to hardware of the processor1910. In addition to a data bus, the bus system1930may further include a power bus, a control bus, a state signal bus, and the like. However, for clarity of description, various buses are marked as the bus system1930in the figure.

Optionally, in another embodiment, the data is an ID type string, the application operation includes the equivalent comparison, and the processor1910performs the equivalent comparison of the data according to the code value.

Specifically, in the another embodiment, the processor1910determines, when the code value is equal to a to-be-compared code value, that the data and data corresponding to the to-be-compared code value are the same data.

Alternatively, in another embodiment, the data is an ID type string or a letter string of a field, the application operation includes sorting, and the processor1910performs the sorting of the data according to the code value.

Specifically, in the another embodiment, the processor1910determines a location of the code value in a to-be-sorted code value according to the code value. The location of the code value is used to represent a location of the data in data corresponding to the to-be-sorted code value.

Alternatively, in another embodiment, the application operation includes the fuzzy search, and the processor1910performs the fuzzy search on the data according to the code value.

Specifically, in the another embodiment, the processor1910determines, according to whether the code value falls within an encoding interval of a prefix string that needs to be fuzzily searched, whether the data includes the prefix string. When the code value falls within the encoding interval of the prefix string that needs to be fuzzily searched, the data includes the prefix string, or when the code value falls outside the encoding interval of the prefix string that needs to be fuzzily searched, the data does not include the prefix string.

It should be understood that the data processing device inFIG. 19is corresponding to the data processing method inFIG. 6, the data processing device inFIG. 19can implement processes of the data processing method inFIG. 6, and for a function of the data processing device inFIG. 19, refer to relevant description of the data processing method inFIG. 6. To avoid repetition, details are not described herein again.

It should be understood that “an embodiment” or “an embodiment” mentioned in this specification means that embodiment-related specific features, structures, or characters are included in at least one embodiment of the present invention. Therefore, “in an embodiment” or “in an embodiment” occurring in the entire specification not necessarily refers to the same embodiment. In addition, these specific features, structures, or characters may be combined in one or more embodiments in any appropriate manner. It should be understood that sequence numbers of the foregoing processes do not mean execution sequences in various embodiments of the present invention. The execution sequences of the processes should be determined according to functions and internal logic of the processes, and should not be construed as any limitation on the implementation processes of the embodiments of the present invention.

It should be understood that in the embodiments of the present invention, “B corresponding to A” indicates that B is associated with A, and B may be determined according to A. However, it should further be understood that determining A according to B does not mean that B is determined according to A only; that is, B may also be determined according to A and/or other information.

With descriptions of the foregoing embodiments, a person skilled in the art may clearly understand that the present invention may be implemented by hardware, firmware or a combination thereof. When the present invention is implemented by software, the foregoing functions may be saved in a computer-readable medium or transmitted as one or more instructions or code in the computer-readable medium. The computer-readable medium includes a computer storage medium and a communications medium, where the communications medium includes any medium that enables a computer program to be transmitted from one place to another. The storage medium may be any available medium accessible to a computer. The following provides an example but does not impose a limitation: The computer-readable medium may include a RAM, a ROM, an EEPROM, a CD-ROM, or another optical disc storage or disk storage medium, or another magnetic storage device, or any other medium that can carry or save expected program code in a form of an instruction or a data structure and can be accessed by a computer. In addition, any connection may be appropriately defined as a computer-readable medium. For example, if software is transmitted from a website, a server or another remote source by using a coaxial cable, an optical fiber/cable, a twisted pair, a digital subscriber line (DSL) or wireless technologies such as infrared ray, radio and microwave, the coaxial cable, optical fiber/cable, twisted pair, DSL or wireless technologies such as infrared ray, radio and microwave are included in fixation of a medium to which they belong. For example, a disk (Disk) and disc (disc) used by the present invention includes a compact disc (CD), a laser disc, an optical disc, a digital versatile disc (DVD), a floppy disk and a Blue-ray disc, where the disk generally copies data by a magnetic means, and the disc copies data optically by a laser means. The foregoing combination should also be included in the protection scope of the computer-readable medium.

In summary, what is described above is merely exemplary embodiments of the technical solutions of the present invention, but is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, or improvement made without departing from the spirit and principle of the present invention shall fall within the protection scope of the present invention.