Package: search-base64
IntroductionIf you have a base64-encoded file and want to find a word into it, you could do this :base64 -d <file> | grep <word>But for a partially base64-encoded file, it is not possible anymore.
With this code, you could do$ echo "HEADER$(echo -n "Hi Bob! How are you doing?" | base64)TRAILER" | python3 -m search_base64.find Bob
-: found Bob => Qm9iInstallationYou can install it as a PyPI package:pip install search-base64Base64Quoting wikipedia, "Base64 is a group of binary-to-text encoding schemes that represent binary data (more specifically, a sequence of 8-bit bytes) in sequences of 24 bits that can be represented by four 6-bit Base64 digits."One of the possible table is the one defined in RFC 4648 §4:IndexBinaryCharIndexBinaryCharIndexBinaryCharIndexBinaryChar0000000A16010000Q32100000g48110000w1000001B17010001R33100001h49110001x2000010C18010010S34100010i50110010y3000011D19010011T35100011j51110011z4000100E20010100U36100100k5211010005000101F21010101V37100101l5311010116000110G22010110W38100110m5411011027000111H23010111X39100111n5511011138001000I24011000Y40101000o5611100049001001J25011001Z41101001p57111001510001010K26011010a42101010q58111010611001011L27011011b43101011r59111011712001100M28011100c44101100s60111100813001101N29011101d45101101t61111101914001110O30011110e46101110u62111110+15001111P31011111f47101111v63111111/Padding is '='.For now, let's assume thata character is encoded as 1 byte. It will be easier to speak about strings instead of bytes.there is only one way to encode a characterHow to get binary representation of a character?Let's use this python command to determine it:f"{ord(char):08b}"Encode "B"For 'B', we get 01000010 so 8 bits.Notice the little gap every 6 bits as they will be used as index to the encoded characterb01b02b03b04b05b06b07b08b09b10b11b12b13b14b15b16b17b18b19b20b21b22b23b2401000010But, by definition, we need 24 digits. So we need to pad the last 16 bits with zeroes.b01b02b03b04b05b06b07b08b09b10b11b12b13b14b15b16b17b18b19b20b21b22b23b24010000100000000000000000Now we will use the previous table to encode these 24 bits into 4 characters.b1b2b3b4b5b6Char010000Q100000g000000=000000=Last 2 characters should be encoded as 'A' but as there is only one character to be encoded the last 2 ones are the padding characters.Now, let's examine how to encode a string of 1, 2 and 3 characters.Encode "Bo"For 'B', we still get 01000010 so 8 bits.For 'o', we get 01101111 so 8 more bits.b01b02b03b04b05b06b07b08b09b10b11b12b13b14b15b16b17b18b19b20b21b22b23b240100001001101111But, by definition, we need 24 digits. So we need to pad the last 8 bits with zeroes.b01b02b03b04b05b06b07b08b09b10b11b12b13b14b15b16b17b18b19b20b21b22b23b24010000100110111100000000Now we will use the previous table to encode these 24 bits into 4 characters.b1b2b3b4b5b6Char010000Q100110m1111008000000=Last character should be encoded as 'A' but, as there are two characters to be encoded, the last one is the padding character.Encode "Bob"For 'B', we still get 01000010 so 8 bits.For 'o', we still get 01101111 so 8 more bits.For 'b', we get 01100010 so 8 more bits.b01b02b03b04b05b06b07b08b09b10b11b12b13b14b15b16b17b18b19b20b21b22b23b24010000100110111101100010This time we don't need any padding.Now we will use the previous table to encode these 24 bits into 4 characters.b1b2b3b4b5b6Char010000Q100110m1111019100010iThis time there is no need to replace anything as there was no padding involved.GeneralisationWe can see that a string can be split into 3-characters chunks (so 38=24 bits) which will be independently encoded as 4 characters (so 46=24 bits).As we've seen :"B" will be encoded as "Qg==""Bo" will be encoded as "Qm8=""Bob" will be encoded as "Qm9i"So :"BobB" will be encoded as "Qm9iQg==""BobBo" will be encoded as "Qm9iQm8=""BobBob" will be encoded as "Qm9iQm9i"We need to do it the other way aroundBut we can't just search for "Qm9i" in the base64-encoded text.As we've just shown, 3 consecutive letters are encoded as base64. The text "Dear Bob!" will be encoded as 3-characters chunks :"Dea" encoded as "RGVh""r B" encoded as "ciBC""ob!" encoded as "b2Ih"As you can notice, "Qm9i" is nowhere to be found even if "Bob" is part of the (clear) text.Indeed, "Bob" can appear in 2 separated 3-characters chunks :x"Bo", "b"yzxy"B", "ob"zwith x, y, z being whatever character (remember: we assume, for now, that any given character is 1 byte).**Let's analyse that first case x"Bo", "b"yz **First part x"Bo"For x, we get abcdefgh so 8 bits. Each letter a to h is either 0 or 1.For 'B', we still get 01000010 so 8 more bits.For 'o', we still get 01101111 so 8 more bits.b01b02b03b04b05b06b07b08b09b10b11b12b13b14b15b16b17b18b19b20b21b22b23b24abcdefgh0100001001101111Yet again, as we already have 24 bits, no padding is required.Now we will use the previous table to encode these 24 bits into 4 characters.b1b2b3b4b5b6CharabcdefSee below 1.gh0100See below 2.001001J101111vSo now we have :one character we don't care about as it does not contain any information about "Bob"one character than can be one of the 400 0100, so 'E'01 0100, so 'U'10 0100, so 'k'11 0100, so '0'So looking for "Bo"('b' is coming thereafter) into the clear text is "equivalent" (so far) to look for "[EUk0]Jv" into the base64-encoded text.Second part "b"yzFor 'b', we still get 01100010 so 8 bits.For y, we get abcdefgh so 8 more bits. Each letter a to h is either 0 or 1.For z, we get ijklmnop so 8 more bits. Each letter i to p is either 0 or 1.b01b02b03b04b05b06b07b08b09b10b11b12b13b14b15b16b17b18b19b20b21b22b23b2401100010abcdefghijklmnopNow we will use the previous table to encode these 24 bits into 4 characters.b1b2b3b4b5b6Char011000Y10abcdSee below 2.efghijSee below 1.klmnopSee below 1.So now we have :two trailing characters we don't care about as they do not contain any information about "Bob"one character than can be one of the 1610 0000, so 'g'10 0001, so 'h'10 0010, so 'i'10 0011, so 'j'10 0100, so 'k'10 0101, so 'l'10 0110, so 'm'10 0111, so 'n'10 1000, so 'o'10 1001, so 'p'10 1010, so 'q'10 1011, so 'r'10 1100, so 's'10 1101, so 't'10 1110, so 'u'10 1111, so 'v'So looking for ("Bo")"b" into the clear text is "equivalent" (so far) to look for "Y[g-v]" into the base64-encoded text.Intermediate result for the "Bob" caseSo now we have more information to look for "Bob" directly inside the base64-encoded:"Qm9i" if "Bob" is in the same 3-characters chunk"[EUk0]JvY[g-v]" if "Bo" and "b" are in 2 distinct chunks.Let's analyse that second case xy"B", "ob"zFirst part xy"B"For x, we get abcdefgh so 8 bits. Each letter a to h is either 0 or 1.For y, we get ijklmnop so 8 more bits. Each letter i to p is either 0 or 1.For 'B', we still get 01000010 so 8 more bits.b01b02b03b04b05b06b07b08b09b10b11b12b13b14b15b16b17b18b19b20b21b22b23b24abcdefghijklmnop01000010Yet again, as we already have 24 bits, no padding is required.
Now we will use the previous table to encode these 24 bits into 4 characters.b1b2b3b4b5b6CharabcdefSee below 1.ghijklSee below 1.mnop01See below 2.000010CSo now we have :two leading characters we don't care about as they do not contain any information about "Bob"one character than can be one of the 160000 01, so 'B'0001 01, so 'F'0010 01, so 'J'0011 01, so 'N'0100 01, so 'R'0101 01, so 'V'0110 01, so 'Z'0111 01, so 'd'1000 01, so 'h'1001 01, so 'l'1010 01, so 'p'1011 01, so 't'1100 01, so 'x'1101 01, so '1'1110 01, so '5'1111 01, so '9'So looking for 'B'("ob" is coming thereafter) into the clear text is "equivalent" (so far) to look for "[BFJNRVZdhlptx159]C" into the base64-encoded text.Second part "ob"zFor 'o', we still get 01101111 so 8 more bits.For 'b', we still get 01100010 so 8 bits.For z, we get abcdefgh so 8 more bits. Each letter a to h is either 0 or 1.b01b02b03b04b05b06b07b08b09b10b11b12b13b14b15b16b17b18b19b20b21b22b23b240110111101100010abcdefghYet again, as we already have 24 bits, no padding is required.
Now we will use the previous table to encode these 24 bits into 4 characters.b1b2b3b4b5b6Char011011b11011020010abSee below 1.cdefghSee below 2.So now we have :one trailing character we don't care about as it does not contain any information about "Bob"one character than can be one of the 4:0010 00, so 'I'0010 01, so 'J'0010 10, so 'K'0010 11, so 'L'So looking for ('B')"ob" into the clear text is "equivalent" (so far) to look for "b2[IJKL]" into the base64-encoded text.Final result for the "Bob" caseSo now we have all information to look for "Bob" directly inside the base64-encoded:"Qm9i" if "Bob" is in the same 3-characters chunk"[EUk0]JvY[g-v]" if "Bo" and "b" are in 2 consecutive chunks."[BFJNRVZdhlptx159]Cb2[IJKL]" if "B" and "ob" are in 2 consecutive chunks.Let's use this tool$ echo -n "Dear Bob!" | base64 | python3 -m search_base64.find Bob
-: found Bob => BCb2I # See case 3 above
$ echo -n "Dear  Bob!" | base64 | python3 -m search_base64.find Bob
-: found Bob => Qm9i # See case 1 above
$ echo -n "Dear   Bob!" | base64 | python3 -m search_base64.find Bob
-: found Bob => EJvYi # See case 2 aboveLet's talk about encodingSo far we assumed that one character was encoded as one byte. But Japanese people will disagree with us.Look at how different is the result to base64-encoded 3 characters :$ echo -n "abc" | base64
YWJj            #  4 b64 characters => 24 bits => 3*8 bits => 3 bytes
$ echo -n "日本語" | base64
5pel5pys6Kqe    # 12 b64 characters => 72 bits => 9*8 bits => 9 bytesThus we need to be careful to count the bytes of the characters, not the characters directly.TODOFinish this READMESpecify encoding of contentSpecify an encoding tableMore testsOptimizeRegex?Non-printable character?