Project: ShardingSphere (App Definition and Development / Database)
File: 2022_12_28_Fuzzy_query_for_CipherColumn__ShardingSphere_5.3.0_Deep_Dive.en.md

+++ title = "Fuzzy query for CipherColumn | ShardingSphere 5.3.0 Deep Dive" weight = 84 chapter = true +++ supports data encryption. By parsing users' SQL input and rewriting the SQL according to the users' encryption rules, the original data is encrypted and stored with ciphertext data in the underlying database at the same time. When a user queries the data, it only fetches the ciphertext data from the database, decrypts it, and finally returns the decrypted original data to the user. However, because the encryption algorithm encrypts the whole string, fuzzy queries cannot be achieved. Nevertheless, many businesses still need fuzzy queries after the data is encrypted. , Apache ShardingSphere provides users with a default fuzzy query algorithm, supporting the fuzzy query for encrypted fields. The algorithm also supports hot plugging, which can be customized by users, and the fuzzy query can be achieved through configuration. Load all the data into the IMDB to decrypt it; then it'll be like querying the original data. This method can achieve fuzzy queries. If the amount of data is small, this method will prove to be simple and cost-effective, while on the other hand, if the amount of data is large, it'll turn out to be a disaster. The second method is to modify fuzzy query conditions and use the database decryption function to decrypt data first and then implement fuzzy query. This method's advantage is the low implementation & development cost, as well as use cost. Users only need to slightly modify the previous fuzzy query conditions. However, the ciphertext and encryption functions are stored together in the database, which cannot cope with the problem of account data leaks. ```sql Native SQL: select * from user where name like "%xxx%" After implementing the decryption function: lt * frm ur whr dd(nam) lk "%%" ``` Implement data masking on ciphertext and then store it in a fuzzy query column. This method could lack in terms of precision. ```markdown For example, mobile number 13012345678 becomes 1305678 after the masking algorithm is performed. ``` This method performs tokenization and combination on ciphertext data and then encrypts the resultset by grouping characters with fixed length and splitting a field into multiple ones. For example, we take four English characters and two Chinese characters as a query condition: `ningyu1` uses the 4-character as a group to encrypt, so the first group is `ning`, the second group `ingy`, the third group `ngyu`, the fourth group `gyu1`, and so on. All the characters are encrypted and stored in the fuzzy query column. If you want to retrieve all data that contains four characters, such as `ingy`, encrypt the characters and use a key `like"%partial%"` to query. Shortcomings: Increased storage costs: free grouping will increase the amount of data and the data length will increase after being encrypted. Limited length in fuzzy query: due to security issues, the length of free grouping cannot be too short, otherwise it will be easily cracked by the . Like the example I mentioned above, the length of fuzzy query characters must be greater than or equal to 4 letters/digits, or 2 Chinese characters. Although the above methods are all viable, it's only natural to wonder if there's a better alternative out there. In our community, we find that single-character encryption and storage can balance both performance and query, but fails to meet security

Then what's the ideal solution? Inspired by masking algorithms and cryptographic hash functions, we find that data loss and one-way functions can be used. The cryptographic hash function should have the following four features: For any given message, it should be easy to calculate the hash value. It should be difficult to infer the original message from a known hash value. It should not be feasible to modify the message without changing the hash value. There should only be a very low chance that two different messages produce the same hash value. Security: because of the one-way function, it's not possible to infer the original message. In order to improve the accuracy of the fuzzy query, we want to encrypt a single character, but it will be cracked by the rainbow table. So we take a one-way function (to make sure every character is the same after encryption) and increase the frequency of collisions (to make sure every string is 1: N backward), which greatly enhances security. Apache ShardingSphere implements a universal fuzzy query algorithm by using the below single-character digest algorithm `org.apache.shardingsphere.encrypt.algorithm.like.CharDigestLikeEncryptAlgorithm`. ```java public final class CharDigestLikeEncryptAlgorithm implements LikeEncryptAlgorithm<Object, String> { private static final String DELTA = "delta"; private static final String MASK = "mask"; private static final String START = "start"; private static final String DICT = "dict"; private static final int DEFAULT_DELTA = 1; private static final int DEFAULTMASK = 0b11110111_1101; private static final int DEFAULT_START = 0x4e00; private static final int MAXNUMERICLETTER_CHAR = 255; @Getter private Properties props; private int delta; private int mask; private int start; private Map<Character, Integer> charIndexes; @Override public void init(final Properties props) { this.props = props; delta = createDelta(props); mask = createMask(props); start = createStart(props); charIndexes = createCharIndexes(props); } private int createDelta(final Properties props) { if (props.containsKey(DELTA)) { String delta = props.getProperty(DELTA); try { return Integer.parseInt(delta); } catch (NumberFormatException ex) { throw new EncryptAlgorithmInitializationException("CHARDIGESTLIKE", "delta can only be a decimal number"); } } return DEFAULT_DELTA; } private int createMask(final Properties props) { if (props.containsKey(MASK)) { String mask = props.getProperty(MASK); try { return Integer.parseInt(mask); } catch (NumberFormatException ex) { throw new EncryptAlgorithmInitializationException("CHARDIGESTLIKE", "mask can only be a decimal number"); } } return DEFAULT_MASK; } private int createStart(final Properties props) { if (props.containsKey(START)) { String start = props.getProperty(START); try { return Integer.parseInt(start); } catch (NumberFormatException ex) { throw new EncryptAlgorithmInitializationException("CHARDIGESTLIKE", "start can only be a decimal number"); } } return DEFAULT_START; } private Map<Character, Integer> createCharIndexes(final Properties props) { String dictContent = props.containsKey(DICT) && !Strings.isNullOrEmpty(props.getProperty(DICT)) ? props.getProperty(DICT) : initDefaultDict(); Map<Character, Integer> result = new HashMap<>(dictContent.length(), 1); for (int index = 0; index < dictContent.length(); index++) { result.put(dictContent.charAt(index), index); } return result; } @SneakyThrows private String initDefaultDict() { InputStream inputStream = CharDigestLikeEncryptAlgorithm.class.getClassLoader().getResourceAsStream("algorithm/like/commonchinesecharacter.dict"); LineProcessor<String> lineProcessor = new LineProcessor<String>() { private final StringBuilder builder = new StringBuilder(); @Override public boolean processLine(final String line) { if (line.startsWith("#") || 0 == line.length()) { return true; } else { builder.append(line); return false; } } @Override public String getResult() { return builder.toString(); } }; return CharStreams.readLines(new InputStreamReader(inputStream, Charsets.UTF_8), lineProcessor); } @Override public String encrypt(final Object plainValue, final EncryptContext encryptContext) { return null == plainValue ? null : digest(String.valueOf(plainValue)); } private String digest(final String plainValue) { StringBuilder result = new StringBuilder(plainValue.length()); for (char each : plainValue.toCharArray()) { char maskedChar = getMaskedChar(each); if ('%' == maskedChar) { result.append(each); } else { result.append(maskedChar); } } return

} private char getMaskedChar(final char originalChar) { if ('%' == originalChar) { return originalChar; } if (originalChar <= MAXNUMERICLETTER_CHAR) { return (char) ((originalChar + delta) & mask); } if (charIndexes.containsKey(originalChar)) { return (char) (((charIndexes.get(originalChar) + delta) & mask) + start); } return (char) (((originalChar + delta) & mask) + start); } @Override public String getType() { return "CHARDIGESTLIKE"; } } ``` Define the binary `mask` code to lose precision `0b111101111101` (mask). Save common Chinese characters with disrupted order like a `map` dictionary. Obtain a single string of `Unicode` for digits, English, and Latin. Obtain `index` for a Chinese character belonging to a dictionary. Other characters fetch the `Unicode` of a single string. Add `1 (delta)` to the digits obtained by different types above to prevent any original text from appearing in the database. Then convert the offset `Unicode` into binary and perform the `AND` operation with `mask`, and carry out a 2-bit digit loss. Directly output digits, English, and Latin after the loss of precision. The remaining characters are converted to decimal and output with the common character `start` code after the loss of precision. Simply use `Unicode` and `mask` code of common characters to perform the `AND` operation. ```yaml Mask: 0b11111111111001111101 The original character: 0b1000101110101111 After encryption: 0b1000101000101101 Assuming we know the key and encryption algorithm, the original string after a backward pass is:1.0b1000101100101101 2.0b1000101100101111 3.0b1000101110101101 4.0b1000101110101111 5.0b1000101010101101 6.0b1000101010101111 7.0b1000101000101111 8.0b1000101000101101 ``` We find that based on the missing bits, each string can be derived `2^n` Chinese characters backward. When the `Unicode` of common Chinese characters is decimal, their intervals are very large. Notice that the Chinese characters inferred backward are not common characters, and it's more likely to infer the original characters. Since the interval of common Chinese characters `Unicode` is irregular, we planned to leave the last few bits of Chinese characters `Unicode` and convert them into decimal as `index` to fetch some common Chinese characters. This way, when the algorithm is known, uncommon characters won't appear after a backward pass, and distractors are no longer easy to eliminate. If we leave the last few bits of Chinese characters `Unicode`, it has something to do with the relationship between the accuracy of fuzzy query and anti-decryption complexity. The higher the accuracy, the lower the decryption difficulty. Let's take a look at the collision degree of common Chinese characters under our algorithm: When `mask`=0b001111111111: When `mask`=0b000111111111: For the mantissa of Chinese characters, leave 10 and 9 digits. The 10-digit query is more accurate because its collision is much weaker. Nevertheless, if the algorithm and the key are known, the original text of the 1:1 character can be derived backward. The 9-digit query is less accurate because 9-digit collisions are relatively stronger, but there are fewer 1:1 characters. We find that although we change the collisions regardless of whether we leave 10 or 9 digits, the distribution is very unbalanced due to the irregular Unicode of Chinese characters, and the overall collision probability cannot be controlled. In response to the unevenly distributed problem found in the second edition, we take common characters with disrupted order as the dictionary table. The encrypted text first looks up the `index` in the out-of-order dictionary table. We use the `index` and subscript to replace the `Unicode` without rules. Use `Unicode` in case of uncommon characters. (Note: evenly distribute the code to be calculated as far as possible.) The next step is to perform the `AND` operation with `mask` and lose 2-bit precision to increase the frequency of

Let's take a look at the collision degree of common Chinese characters under our algorithm: When `mask`=0b111110111101: When `mask`=0b011110111101: When the `mask` leaves 11 bits, you can see that the collision distribution is concentrated at 1:4. When `mask` leaves 10 bits, the number becomes 1:8. At this time, we only need to adjust the number of precision losses to control whether the collision is 1:2, 1:4 or 1:8. If `mask` is selected as 1, and the algorithm and key are known, there will be a 1:1 Chinese character, because what we calculate at this time is the collision degree of common characters. If we add the missing 4 bits before the 16-bit binary of Chinese characters, the situation becomes `2^5=32` cases. Since we encrypted the whole text, even if the individual character is inferred backwards, there will be little impact on overall security, and it will not cause mass data leaks. At the same time, the premise of backward pass is to know the algorithm, key, `delta` and dictionary, so it's impossible to achieve from the data in the database. Fuzzy query requires the configuration of `encryptors`(encryption algorithm configuration), `likeQueryColumn` (fuzzy query column name), and `likeQueryEncryptorName`(encryption algorithm name of fuzzy query column ) in the encryption configuration. Please refer to the following configuration. Add your own sharding algorithm and data source. ```yaml dataSources: ds_0: dataSourceClassName: com.zaxxer.hikari.HikariDataSource driverClassName: com.mysql.jdbc.Driver jdbcUrl: jdbc:mysql://127.0.0.1:3306/test?allowPublicKeyRetrieval=true username: root password: root rules: !ENCRYPT encryptors: like_encryptor: type: CHARDIGESTLIKE aes_encryptor: type: AES props: aes-key-value: 123456abc tables: user: columns: name: cipherColumn: name encryptorName: aes_encryptor assistedQueryColumn: name_ext assistedQueryEncryptorName: aes_encryptor likeQueryColumn: name_like likeQueryEncryptorName: like_encryptor phone: cipherColumn: phone encryptorName: aes_encryptor likeQueryColumn: phone_like likeQueryEncryptorName: like_encryptor props: sql-show: true ``` Insert ```sql Logic SQL: insert into user ( id, name, phone, sex) values ( 1, '', '13012345678', '') Actual SQL: ds0 ::: insert into user ( id, name, nameext, namelike, phone, phonelike, sex) values (1, 'gyVPLyhIzDIZaWDwTl3n4g==', 'gyVPLyhIzDIZaWDwTl3n4g==', '', 'qEmE7xRzW0d7EotlOAt6ww==', '04101454589', '') ``` Update ```sql Logic SQL: update user set name = '123', sex = '1' where sex ='' and phone like '130%' Actual SQL: ds0 ::: update user set name = 'K22HjufsPPy4rrf4PD046A==', nameext = 'K22HjufsPPy4rrf4PD046A==', namelike = '014', sex = '1' where sex ='' and phonelike like '041%' ``` Select ```sql Logic SQL: select * from user where (id = 1 or phone = '13012345678') and name like '%' Actual SQL: ds0 ::: select `user`.`id`, `user`.`name` AS `name`, `user`.`sex`, `user`.`phone` AS `phone`, `user`.`createtime` from user where (id = 1 or phone = 'qEmE7xRzW0d7EotlOAt6ww==') and name_like like '%' ``` Select: federated table sub-query ```sql Logic SQL: select * from user LEFT JOIN userext on user.id=userext.id where user.id in (select id from user where sex = '' and name like '%') Actual SQL: ds0 ::: select `user`.`id`, `user`.`name` AS `name`, `user`.`sex`, `user`.`phone` AS `phone`, `user`.`createtime`, `userext`.`id`, `userext`.`address` from user LEFT JOIN userext on user.id=userext.id where user.id in (select id from user where sex = '' and name_like like '%') ``` Delete ```sql Logic SQL: delete from user where sex = '' and name like '%' Actual SQL: ds0 ::: delete from user where sex = '' and namelike like '%' ``` The above example demonstrates how fuzzy query columns rewrite SQL in different SQL syntaxes to support fuzzy queries. This blog post introduced you to the working principles of fuzzy query and used specific examples to demonstrate how to use it. We hope that through this article, you will have gained a basic understanding of fuzzy queries. Xiong Gaoxiang, an engineer at and a ShardingSphere Contributor, is responsible for the data encryption and data masking R&D.