Python Tokens and Character Sets

07 Feb, 2022
Python is a general-purpose, high-level programming language. It was designed with an emphasis on code readability, and its syntax allows programmers to express their concepts in fewer lines of code, and these codes are known as scripts. These scripts contain character sets, tokens, and identifiers. In this article, we will learn about these character sets, tokens, and identifiers.
A character set is a set of valid characters acceptable by a programming language in scripting. In this case, we are talking about the Python programming language. So, the Python character set is a valid set of characters recognized by the Python language. These are the characters we can use during writing a script in Python. Python supports all ASCII / Unicode characters that include:
Alphabets: All capital (A-Z) and small (a-z) alphabets.
Digits: All digits 0-9.
Special Symbols: Python supports all kind of special symbols like, ” ‘ l ; : ! ~ @ # $ % ^ ` & * ( ) _ + – = { } [ ] \ .
White Spaces: White spaces like tab space, blank space, newline, and carriage return.
Other: All ASCII and UNICODE characters are supported by Python that constitutes the Python character set.
A token is the smallest individual unit in a python program. All statements and instructions in a program are built with tokens. The various tokens in python are :
1. Keywords: Keywords are words that have some special meaning or significance in a programming language. They can’t be used as variable names, function names, or any other random purpose. They are used for their special features. In Python we have 33 keywords some of them are: try, False, True, class, break, continue, and, as, assert, while, for, in, raise, except, or, not, if, elif, print, import, etc.
Python3
# for loopfor x in range(1, 9):         # Print the value of x    print(x)         # Check if the value of x is less than 6    # Here if the value of x is less than 6    # then the loop will continue    # Here, if, continue, else, break,    # for loop are keywords    if x < 6:        continue         # If i greater then 6 then break loop    else:        break
Output:
1
2
3
4
5
6
2. Identifiers: Identifiers are the names given to any variable, function, class, list, methods, etc. for their identification. Python is a case-sensitive language and it has some rules and regulations to name an identifier. Here are some rules to name an identifier:-
As stated above, Python is case-sensitive. So case matters in naming identifiers. And hence geeks and Geeks are two different identifiers.
Identifier starts with a capital letter (A-Z) , a small letter (a-z) or an underscore( _ ). It can’t start with any other character.
Except for letters and underscore, digits can also be a part of identifier but can’t be the first character of it.
Any other special characters or whitespaces are strictly prohibited in an identifier.
An identifier can’t be a keyword.
For Example: Some valid identifiers are gfg, GeeksforGeeks, _geek, mega12, etc. While 91road, #tweet, i am, etc. are not valid identifiers.
Python3
# Here GFG and b are the identifierGFG = 'Hello'b = "Geeks" # Driver codeprint(GFG)print(b)
Output:
Hello
Geeks
3. Literals or Values: Literals are the fixed values or data items used in a source code. Python supports different types of literals such as:
(i) String Literals: The text written in single, double, or triple quotes represents the string literals in Python. For example: “Computer Science”, ‘sam’, etc. We can also use triple quotes to write multi-line strings.
Python3
# String Literalsa = 'Hello'b = "Geeks"c = '''Geeks for Geeks is a        learning platform''' # Driver codeprint(a)print(b)print(c)
Hello
Geeks
Geeks for Geeks is a 
        learning platform
(ii) Character Literals: Character literal is also a string literal type in which the character is enclosed in single or double-quotes.
Python3
# Character Literalsa = 'G'b = "W" # Driver codeprint(a)print(b)
Output:
G
W
(iii) Numeric Literals: These are the literals written in form of numbers. Python supports the following numerical literals:
Integer Literal: It includes both positive and negative numbers along with 0. It doesn’t include fractional parts. It can also include binary, decimal, octal, hexadecimal literal.
Float Literal: It includes both positive and negative real numbers. It also includes fractional parts.
Complex Literal: It includes a+bi numeral, here a represents the real part and b represents the complex part.
Python3
# Numeric Literalsa = 5b = 10.3c = -17 # Driver codeprint(a)print(b)print(c)
5
10.3
-17
(iv) Boolean Literals: Boolean literals have only two values in Python. These are True and False.
Python3
# Boolean Literalsa = 3b = (a == 3)c = True + 10 # Driver codeprint(a, b, c)
3 True 11
(v) Special Literals: Python has a special literal ‘None’. It is used to denote nothing, no values, or the absence of value.
Python3
# Special Literalsvar = Noneprint(var)
None
(vi) Literals Collections: Literals collections in python includes list, tuple, dictionary, and sets.
List: It is a list of elements represented in square brackets with commas in between. These variables can be of any data type and can be changed as well.Tuple: It is also a list of comma-separated elements or values in round brackets. The values can be of any data type but can’t be changed.Dictionary: It is the unordered set of key-value pairs.Set: It is the unordered collection of elements in curly braces ‘{}’.
List: It is a list of elements represented in square brackets with commas in between. These variables can be of any data type and can be changed as well.
Tuple: It is also a list of comma-separated elements or values in round brackets. The values can be of any data type but can’t be changed.
Dictionary: It is the unordered set of key-value pairs.
Set: It is the unordered collection of elements in curly braces ‘{}’.
Python3
# Literals collections# Listmy_list = [23, "geek", 1.2, 'data'] # Tuplemy_tuple = (1, 2, 3, 'hello')   # Dictionarymy_dict = {1:'one', 2:'two', 3:'three'}   # Setmy_set = {1, 2, 3, 4}  # Driver codeprint(my_list)print(my_tuple)print(my_dict)print(my_set)
[23, 'geek', 1.2, 'data']
(1, 2, 3, 'hello')
{1: 'one', 2: 'two', 3: 'three'}
{1, 2, 3, 4}
4. Operators: These are the tokens responsible to perform an operation in an expression. The variables on which operation is applied are called operands. Operators can be unary or binary. Unary operators are the ones acting on a single operand like complement operator, etc. While binary operators need two operands to operate.
Python3
# Operatorsa = 12  # Unary operatorb = ~ a # Binary operatorc = a+b   # Driver codeprint(b)print(c)
-13
-1
5. Punctuators: These are the symbols that used in Python to organize the structures, statements, and expressions. Some of the Punctuators are: [ ] { } ( ) @  -=  +=  *=  //=  **==  = , etc.
saurabh1990aror
Picked
Class 11
Python
School Learning
School Programming
Writing code in comment?
Please use ide.geeksforgeeks.org,
generate link and share the link here.
Geometric Series
Cyber Forensics
Impact of Technology on Society
Causes and Effects of Water Pollution
Statistics
Read JSON file using Python
Adding new column to existing DataFrame in Pandas
Python map() function
How to get column names in Pandas dataframe