Byte string, Unicode string, Raw string — A Guide to all strings in Python | by Guangyuan(Frank) Li | Towards Data Science

“String” in Python? Sounds like the most basic topics that every Python programmer should have already mastered in their first Python tutorial. However, do you know there are at least four types of strings in primitive Python? Do you know how your strings are actually represented in Numpy or Pandas or any other packages? What are the differences and caveats that I need to know? (See below)
Here, Let me try to clear some of your confusion based on my own learning experiences. We are going to cover these topics:
What are the concepts of “Encoding” and “Decoding”?What is a raw(r) string or format(f) string and when I should use them?What are the differences between Numpy/Pandas string and primitive Python strings?
What are the concepts of “Encoding” and “Decoding”?
What is a raw(r) string or format(f) string and when I should use them?
What are the differences between Numpy/Pandas string and primitive Python strings?
To understand the differences between byte string and Unicode string, we first need to know what “Encoding” and “Decoding” are.
To store the human-readable characters on computers, we need to encode them into bytes. In contrast, we need to decode the bytes into human-readable characters for representation. Byte, in computer science, indicates a unit of 0/1, commonly of length 8. So characters “Hi” are actually stored as “01001000 01101001” on the computer, which consumes 2 bytes (16-bits).
The rule that defines the encoding process is called encoding schema, commonly used ones include “ASCII”, “UTF-8”, etc. Now, the question how do these encoding schemas look like?
“ASCII” converts each character into one byte. Since one byte consisted of 8 bits and each bit contains 0/1. The total number of characters “ASCII” can represent is 28=256. It is more than enough for 26 English letters plus some commonly-used characters. See the “ASCII” table for full information.
However, 256 characters are obviously not enough for storing all the characters in the world. In light of that, people designed Unicode in which each character will be encoded as a “code point”. For instance, “H” will be represented as code point “U+0048”. According to Wikipedia, Unicode can include 144,697 characters. But again, the code point still can not be recognized by the computer, so we have “UTF-8” or other variants encoding schema to convert the code point to the byte. “UTF-8” means the minimum length of bits to represent a character is 8, so you can guess, “UTF-16” means the minimum length of bits is 16. UTF-8 is way more popular than UTF-16 so in this article and for most of your work as they are compatible with the old original ASCII standard (one character can be represented using one byte), understanding the UTF-8 is enough. See the “UTF-8” table for full information.
With the basic concepts understood, let’s cover some practical coding tips in Python. In Python3, the default string is called Unicode string (u string), you can understand them as human-readable characters. As explained above, you can encode them to the byte string (b string), and the byte string can be decoded back to the Unicode string.
u'Hi'.encode('ASCII')> b'Hi'b'\x48\x69'.decode('ASCII')> 'Hi'
In Python IDE, usually, the byte string will be automatically decoded using “ASCII” when printed out, so that’s why the first result is human-readable (b’Hi'). More often, Byte string should be represented as hex code (b’\x48\x69'), which can be found in any “ASCII” table.
To wrap up this section, let’s look at one “UTF-8” example, again the hex code for every character can be found in the UTF-8 table:
b'\xe0\xb0\x86'.decode('utf-8')> 'ఆ'
To start with this type of string, we just need to know one thing about the default Unicode string (u string) — backslash (“\”) is a special character in Unicode string such that the following character will have the special meanings (i.e. \t, \n, etc). So in order to ignore the special meaning of the backslash, we have the Raw string (r string) in which backslash is just a backslash and it won’t have effects on changing the meaning of its following characters.
Here comes my personal suggestions, unless in the scenario where you need to define the regular expression match pattern (See below example), I suggest using the Unicode string with escape (using backslash to ignore special character). As shown in the third example, we used backslash to make sure we output a literal “\” instead of a new tab “\t”.
Why I would recommend that? This is because the raw string can not really solve everything, for instance, how to output a literal single quotation in a Raw string?
r'ttt'g''File "<ipython-input-76-2839752ff4e6>", line 1    r'ttt'g''          ^SyntaxError: invalid syntax
However, using the escape idea along with Unicode string seems to be a more general approach:
u'ttt\'g\''> "ttt'g'"
The only place that Raw string (r string) may be useful is when you are dealing with regular expression. The regular expression is a whole can of worms and I am not intending to cover that in this article. But when using a regular expression, we usually need to first define a matched pattern where the Raw string would be recommended.
import repat = re.compile(r'ENSG\d+$')string = 'ENSG00000555're.search(pat,string)<_sre.SRE_Match object; span=(0, 12), match='ENSG00000555'>
For experienced Python programmers, format string should not be an unfamiliar concept, it allows you to dynamically configure the string we want to print. Before Python version 3.5, the recommended approach for creating format string is like that:
var = 'hello'print('{} world'.format(var))> hello world
Since Python 3.5 and later, there’s a new “f string” to help us to achieve the same goal:
var = 'hello'print(f'{var} world')> hello world
The important thing I want to note here is, when using format string, curly brace “{}” becomes a very special character and contains its unique meaning. As a result, if we still aim to output the literal “{}”, we need to escape it use double curly brace “{{}}”:
'{{}}{}'.format(5)> '{}5'
What we covered so far are all about primitive string types in Python, we haven’t touched on how the string is handled in other popular Python packages. Here I am going to share a bit on string types in Numpy and Pandas.
In Numpy, usually, String can be specified in three different “dtypes”:
Variable-length Unicode (U)Fixed-length byte (S)Python object (O)
Variable-length Unicode (U)
Fixed-length byte (S)
Python object (O)
import numpy as nparr1 = np.array(['hello','hi','ha'],dtype='<U5')arr2 = np.array(['hello','hi','ha'],dtype='|S5')arr3 = np.array(['hello','hi','ha'],dtype='object')> array(['hello', 'hi', 'ha'], dtype='<U5')> array([b'hello', b'hi', b'ha'], dtype='|S5')> array(['hello', 'hi', 'ha'], dtype=object)
<U5 means the longest one is a string of length 5, however, a more memory-saving way is to use fixed-length |S5 which essentially convert them to byte strings. This is the preferred way if you are trying to convert it to a strong-typed data structure (i.e. store as h5 file). In addition, we can treat strings as Python objects and store them in the Numpy array, since every Python object can be stored in the Numpy array using the “object” type.
String in pandas can be represented in two ways:
object dtype (most of the time should be fine)string dtype
object dtype (most of the time should be fine)
string dtype
import pandas as pds1 = pd.Series(['hello','hi','ha'],dtype='object')s2 = pd.Series(['hello','hi','ha'],dtype='string')> s10    hello1       hi2       hadtype: object> s20    hello1       hi2       hadtype: string
These two types are in general similar, the subtle differences are outlined in the documentation.
In summary, we talked about the different representations of “string” in Python. Starting with the default Unicode string (u string), we touched on how it relates to Byte string (b string). Understanding the conversion is very important because sometimes the standard output from other programs will be in the format of bytes, and we need to first decode them to Unicode string for further Streaming operation. We then talked about Raw string (r string) and Format string (f string) and the caveats we need to pay attention to when using them. Finally, we summarised the different ways of string representation in Numpy and Pandas, and special care should be taken when instantiating Numpy or Pandas objects with string because the behaviors will be drastically different than primitive Python strings.
That’s about it! I hope you find this article interesting and useful, thanks for reading! If you like this article, follow me on medium, thank you so much for your support. Connect me on my Twitter or LinkedIn, also please let me know if you have any questions or what kind of tutorials you would like to see in the future!