# Python Concepts/List Comprehension
## Objective

* To understand and use List Comprehensions, including Nested List Comprehensions.

## Lesson

* To create a list of squares of numbers in range(1,6)
* we can do something like:
* >>> a =
* >>> for x in range(1,6) : a += [x*x]
* >>> a
* [1, 4, 9, 16, 25]
* This approach has the perhaps undesirable side-effect of creating or re-assigning both
* a and x. It would be convenient if we could do something like:
* [for x in range(1,6) : [x*x]] # This produces SyntaxError: invalid syntax.
* If we change the syntax slightly, List Comprehensions come to the rescue.

## List Comprehensions

* List comprehensions provide a concise way to create lists. Common applications are to make new lists where each element is the result of some operations applied to each member of another sequence or iterable, or to create a subsequence of those elements that satisfy a certain condition.
* Consider the task mentioned above, to create a list of squares of numbers in range(1,6). Modify the syntax slightly, and put it into the form of a List Comprehension or "listcomp":
* >>> [x*x for x in range(1,6)]
* [1, 4, 9, 16, 25]
* In the listcomp above, x is local to the listcomp, and the final bracket ']' tells python that it should expect no more input.
* A list comprehension consists of brackets containing an expression followed by a for clause, then zero or more for or if clauses. The result will be a new list. Consider some examples:
* Given:
* >>> a = list(range(-5,7)) ; a
* [-5, -4, -3, -2, -1, 0, 1, 2, 3, 4, 5, 6]
* Create a deep copy of list a:
* >>> b = [p for p in a] ; b
* [-5, -4, -3, -2, -1, 0, 1, 2, 3, 4, 5, 6]
* >>> a is b
* False # a and b are different entities.
* Display the position and value of all odd numbers in list a:
* >>> [(i,a[i]) for i in range(len(a)) if a[i] % 2]
* [(0, -5), (2, -3), (4, -1), (6, 1), (8, 3), (10, 5)]
* The listcomp above is equivalent to:
* b = [
* ( 0, a[0]),
* #( 1, a[1]), not included. a[1] is even.
* ( 2, a[2]),
* #( 3, a[3]), not included. a[3] is even.
* ( 4, a[4]),
* #( 5, a[5]), not included. a[5] is even.
* ( 6, a[6]),
* #( 7, a[7]), not included. a[7] is even.
* ( 8, a[8]),
* #( 9, a[9]), not included. a[9] is even.
* (10, a[10]),
* #(11, a[11]) not included. a[11] is even.
* print ('b =', b)
* b = [(0, -5), (2, -3), (4, -1), (6, 1), (8, 3), (10, 5)]
* Display all numbers in list a that are an exact integer power of 2:
* >>> [num for num in a for exponent in range(5) if num == 2 ** exponent ]
* [1, 2, 4]
* >>> [ # Same listcomp again with some formatting.
* ... num
* ... for num in a
* ... for exponent in range(5)
* ... if num == 2 ** exponent ] # Between the brackets white space is almost irrelevant.
* [1, 2, 4]
* The above listcomp is equivalent to:
* >>> b =
* >>> a = list(range(-5,7)) ; a
* [-5, -4, -3, -2, -1, 0, 1, 2, 3, 4, 5, 6]
* >>> for num in a :
* ... for exponent in range(5) :
* ... if num == 2 ** exponent : b += [num]
* >>> b
* [1, 2, 4]
* Apply a math function to all elements:
* >>> [(2*x*x + 3*x - 2) for x in a]
* [33, 18, 7, 0, -3, -2, 3, 12, 25, 42, 63, 88]
* Create a list of random positive integers:
* import random
* b = [
* random.getrandbits(p+q-r)
* for p in range(10,20,4)
* for q in range(3,17,5)
* for r in range(4,16,7)
* if ((p+q-r) > 12) and ((p+q-r) % 3)
* print ('b =', b)
* Successive invocations of the above produce:
* b = [1028, 193714, 7439, 6543953, 54358, 116875, 3156193, 644588]
* b = [1665, 267591, 813, 6463934, 35700, 112421, 2943558, 640640]
* b = [4370, 330632, 8045, 5037055, 5526, 44136, 2528805, 612774]
* b = [14764, 21409, 8140, 3436253, 55785, 96832, 1355388, 162655]
* Call a method on each element:
* >>> fruits = [' apples ', ' pears ', ' grapes ', ' WaterMELons ', ' peaches ' ]
* >>> [fruit.strip.lower for fruit in fruits ]
* ['apples', 'pears', 'grapes', 'watermelons', 'peaches']
* To flatten a list:
* >>> L1 = [32, 97, [192, 128], 98, 99, 32, [192, 128], 32]
* >>> [num for elem in L1 for num in elem]
* Traceback (most recent call last):
* File "<stdin>", line 1, in <module>
* File "<stdin>", line 1, in <listcomp>
* TypeError: 'int' object is not iterable
* >>> L1a = [ ([p], p)[isinstance(p, list)] for p in L1 ] ; L1a
* [[32], [97], [192, 128], [98], [99], [32], [192, 128], [32]]
* >>> [num for elem in L1a for num in elem]
* [32, 97, 192, 128, 98, 99, 32, 192, 128, 32]

## Nested List Comprehensions

* Create a useful list:
* matrix = [p for p in range (3,23)] # A listcomp does this nicely.
* print ('len(matrix) =', len(matrix))
* print ('matrix =', matrix)
* len(matrix) = 20
* matrix = [3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22]
* New list of 3 rows
* Create a list containing three rows in which each row contains every third value of matrix.
* b = [ [matrix[p] for p in range(q,len(matrix),3)] for q in range(3) ]
* print ('''
* b =
*, length=
*, length=
*, length=
* '''.format( b[0],len(b[0]), b[1],len(b[1]), b[2],len(b[2]) ))
* b =
* [3, 6, 9, 12, 15, 18, 21], length=7
* [4, 7, 10, 13, 16, 19, 22], length=7
* [5, 8, 11, 14, 17, 20], length=6
* The nested listcomp above is equivalent to:
* b = [ [matrix[p] for p in range(0,len(matrix),3)] ,
* [matrix[p] for p in range(1,len(matrix),3)] ,
* [matrix[p] for p in range(2,len(matrix),3)] ]
* Fill each row as necessary so that all rows have same length:
* c = [ b[p]+[None]*q for p in range(len(b)) for q in [ len(b[0]) - len(b[p]) ] ]
* The syntax of the listcomp requires a for statement. The second for statement
* above assigns a specific value to q, either 0 or 1.
* print ('''
* c =
*, length=
*, length=
*, length=
* '''.format( c[0],len(c[0]), c[1],len(c[1]), c[2],len(c[2]) ))
* c =
* [3, 6, 9, 12, 15, 18, 21], length=7
* [4, 7, 10, 13, 16, 19, 22], length=7
* [5, 8, 11, 14, 17, 20, None], length=7
* Create a dictionary
* Create a dictionary from a list in which each value at an even position is a key and each value at an odd position is the associated value:
* data = [p%q for p in range (3,10) for q in range (2,7)]
* print ('data =', data)
* data = [1, 0, 3, 3, 3, 0, 1, 0, 4, 4, 1, 2, 1, 0, 5, 0, 0, 2, 1, 0, 1, 1, 3, 2, 1, 0, 2, 0, 3, 2, 1, 0, 1, 4, 3]
* Create a list containing two rows in which the first row contains keys and the second contains values.
* b = [ [data[p] for p in range(q,len(data),2)] for q in range(2) ]
* print ('''
* b =
*, length=
*, length=
* '''.format( b[0],len(b[0]), b[1],len(b[1]) ))
* b =
* [1, 3, 3, 1, 4, 1, 1, 5, 0, 1, 1, 3, 1, 2, 3, 1, 1, 3], length=18
* [0, 3, 0, 0, 4, 2, 0, 0, 2, 0, 1, 2, 0, 0, 2, 0, 4], length=17
* Fill as necessary:
* c = [ b[p]+[None]*q for p in range(len(b)) for q in range(len(b[0]) - len(b[p]), 1+len(b[0]) - len(b[p])) ]
* print ('''
* c =
*, length=
*, length=
* '''.format( c[0],len(c[0]), c[1],len(c[1]) ))
* c =
* [1, 3, 3, 1, 4, 1, 1, 5, 0, 1, 1, 3, 1, 2, 3, 1, 1, 3], length=18
* [0, 3, 0, 0, 4, 2, 0, 0, 2, 0, 1, 2, 0, 0, 2, 0, 4, None], length=18
* Transpose rows and columns and create input to dictionary:
* d = [[row[i] for row in c] for i in range(len(c[0]))]
* print (' d =', d);
* d = [[1, 0], [3, 3], [3, 0], [1, 0], [4, 4], [1, 2], [1, 0], [5, 0], [0, 2], [1, 0], [1, 1], [3, 2], [1, 0], [2, 0], [3, 2], [1, 0], [1, 4], [3, None]]
* List d is equivalent to:
* d1 = [ [ c[0][i], c[1][i] ] for i in range(18) ]
* print ('d1 =', d1)
* d1 = [[1, 0], [3, 3], [3, 0], [1, 0], [4, 4], [1, 2], [1, 0], [5, 0], [0, 2], [1, 0], [1, 1], [3, 2], [1, 0], [2, 0], [3, 2], [1, 0], [1, 4], [3, None]]
* List d1 is equivalent to:
* d2 = [
* [ c[0][ 0], c[1][ 0] ] ,
* [ c[0][ 1], c[1][ 1] ] ,
* [ c[0][ 2], c[1][ 2] ] ,
* [ c[0][15], c[1][15] ] ,
* [ c[0][16], c[1][16] ] ,
* [ c[0][17], c[1][17] ]
* print ('d2 =', d2)
* d2 = [[1, 0], [3, 3], [3, 0], [1, 0], [4, 4], [1, 2], [1, 0], [5, 0], [0, 2], [1, 0], [1, 1], [3, 2], [1, 0], [2, 0], [3, 2], [1, 0], [1, 4], [3, None]]
* e = dict(d)
* print ('e =', e);
* e = {1: 4, 3: None, 4: 4, 5: 0, 0: 2, 2: 0}
* Listcomps simplified
* In practice this code will do the job:
* data = [p%q for p in range (3,10) for q in range (2,7)]
* print ('data =', data)
* data = [1, 0, 3, 3, 3, 0, 1, 0, 4, 4, 1, 2, 1, 0, 5, 0, 0, 2, 1, 0, 1, 1, 3, 2, 1, 0, 2, 0, 3, 2, 1, 0, 1, 4, 3]
* Check the input:
* if isinstance(data,list) and len(data) >= 1: pass # Input must contain at least one key.
* else : exit(99)
* status = {
* ((isinstance(data[p], int)) or (isinstance(data[p], float)) or (isinstance(data[p], str)))
* for p in range(0, len(data), 2)
* } # A set comprehension. In this dictionary each key must be int or float or str.
* print ('status =', status)
* status = {True}
* if False in status :
* print ("'data' contains unrecognized key.")
* exit (98)
* Create the dictionary
* b = dict(
* (data+[None])[p:p+2] for p in range(0, len(data), 2)
* print (
* '\nlen(data) =', len(data),
* '\nInput to dict\n =', [ (data+[None])[p:p+2] for p in range(0, len(data), 2) ],
* '\nDictionary =', b)
* len(data) = 35
* Input to dict
* = [[1, 0], [3, 3], [3, 0], [1, 0], [4, 4], [1, 2], [1, 0], [5, 0], [0, 2], [1, 0], [1, 1], [3, 2], [1, 0], [2, 0], [3, 2], [1, 0], [1, 4], [3, None]]
* Dictionary = {1: 4, 3: None, 4: 4, 5: 0, 0: 2, 2: 0}

## List Comprehensions for free-format Python

* Although a listcomp recognizes statements beginning with only for or if,
* with a little dexterity the equivalent of assignments and else statements can be
* contained within a listcomp. The advantage is that a listcomp accepts free-format
* Python.
* A "Unix date" has format
* $ date
* Wed Feb 14 08:24:24 CST 2018
* The code in this section uses list comprehensions to recognize valid dates. All of the following are considered valid dates:
* Wed Feb 14 08:24:24 CST 2018
* Wednes Feb 14 08:24:24 CST 2018 # More than 3 letters in name of day.
* Wed Febru 14 08:24:24 CST 2018 # More than 3 letters in name of month.
* Wed Feb 14 8:24 : 24 CST 2018 # White space in hh:mm:ss.
* wed FeB 14 8:24 : 24 cSt 2018 # Bad punctuation.
* Build dictionary months:
* mo = '''January February March April
* May June July August September
* October November December
* L1 = [
* [ month[:3], {month[:p] for p in range (3,len(month)+1)} ]
* for month in mo.title.split
* months = dict(L1)
* Display dictionary months:
* L1 = [
* '''months[''] ='''.format(key, months[key])
* for key in months
* print ( '\n'.join(L1) )
* months['Jan'] = {'Januar', 'Janua', 'Janu', 'Jan', 'January'}
* months['Feb'] = {'Februa', 'Febru', 'Februar', 'Feb', 'Febr', 'February'}
* months['Mar'] = {'Mar', 'March', 'Marc'}
* months['Apr'] = {'April', 'Apri', 'Apr'}
* months['May'] = {'May'}
* months['Jun'] = {'June', 'Jun'}
* months['Jul'] = {'Jul', 'July'}
* months['Aug'] = {'Augus', 'Augu', 'Aug', 'August'}
* months['Sep'] = {'Sep', 'Septemb', 'September', 'Septem', 'Septe', 'Septembe', 'Sept'}
* months['Oct'] = {'Octo', 'Octobe', 'Oct', 'Octob', 'October'}
* months['Nov'] = {'Nove', 'November', 'Novemb', 'Novem', 'Nov', 'Novembe'}
* months['Dec'] = {'Decembe', 'Decemb', 'Dece', 'Decem', 'December', 'Dec'}
* Build dictionary days:
* da='''
* Sunday Monday Tuesday
* Wednesday Thursday
* Friday Saturday
* L1 = [
* [ day[:3], {day[:p] for p in range (3,len(day)+1)} ]
* for day in da.title.split
* days = dict(L1)
* Display dictionary days:
* L1 = [
* '''days[''] ='''.format(key, days[key])
* for key in days
* print ( '\n'.join(L1) )
* days['Sun'] = {'Sund', 'Sunda', 'Sun', 'Sunday'}
* days['Mon'] = {'Monday', 'Mon', 'Mond', 'Monda'}
* days['Tue'] = {'Tuesda', 'Tues', 'Tuesday', 'Tue', 'Tuesd'}
* days['Wed'] = {'Wednesda', 'Wednesd', 'Wedn', 'Wednes', 'Wed', 'Wedne', 'Wednesday'}
* days['Thu'] = {'Thursday', 'Thur', 'Thu', 'Thursd', 'Thurs', 'Thursda'}
* days['Fri'] = {'Friday', 'Fri', 'Frida', 'Frid'}
* days['Sat'] = {'Saturday', 'Satu', 'Saturda', 'Sat', 'Saturd', 'Satur'}
* The regular expression:
* reg2 = (
* r'''\b # Word boundary.
* (?P<day>\w{3,}) # At least 3 word characters.
* \s+
* (?P<month>\w{3,}) # At least 3 word characters.
* \s+
* (?P<date>\d{1,2}) # 1 or 2 numbers.
* \s+
* (?P<hours>\d{1,2}) # 1 or 2 numbers.
* \s*:\s*
* (?P<minutes>\d{1,2}) # 1 or 2 numbers.
* \s*:\s*
* (?P<seconds>\d{1,2}) # 1 or 2 numbers.
* \s+
* (?P<time_zone>\w{3}) # 3 word characters
* \s+
* (?P<year>\d{4}) # 4 numbers
* \b # Word boundary.'''
* Dictionary that contains number of days per month:
* d1 = dict ((
* ('Jan', 31), ('May', 31), ('Sep', 30),
* ('Feb', 28), ('Jun', 30), ('Oct', 31),
* ('Mar', 31), ('Jul', 31), ('Nov', 30),
* ('Apr', 30), ('Aug', 31), ('Dec', 31),
* List all valid dates in string dates below.
* dates = '''
* MON Februar 12 0:30 : 19 CST 2018
* Tue Feb 33 00:30:19 CST 2018 # Invalid.
* Wed Feb 29 00:30:19 CST 1900 # Invalid.
* Thursda feb 29 00:30:19 CST 1944
* The list comprehension that does it all:
* L1 = [
* '\n'.join(( str(m), m[0], str(m.groupdict) ))
* for m in re.finditer(reg2, dates, re.IGNORECASE|re.VERBOSE|re.ASCII)
* for day in (m['day'].title,) # Equivalent to assignment: day = m['day'].title
* if day[:3] in days
* if day in days[day[:3]]
* for month in ( m['month'].title ,)
* if month[:3] in months
* if month in months[month[:3]]
* for date in ( int(m['date']) ,) if date >= 1
* for hours in ( int(m['hours']) ,) if hours <= 23
* for minutes in ( int(m['minutes']) ,) if minutes <= 59
* for seconds in ( int(m['seconds']) ,) if seconds <= 59
* for zone in (m['time_zone'] ,) if zone.upper in ('EST', 'EDT', 'CST', 'CDT', 'MST', 'MDT', 'PST', 'PDT' )
* for year in ( int(m['year']) ,) if year >= 1900 and year <= 2020
* for leap_year in ( # 'else' in a listcomp
* ( # equivalent to:
* year % 4 == 0, # if year % 100 == 0:
* year % 400 == 0 # leap_year = year % 400 == 0
* )[year % 100 == 0] # else :
* ,) # leap_year = year % 4 == 0
* for max_date in ( # if (month[:3] == 'Feb') and leap_year :
* ( # max_date = 29
* d1[month[:3]], # else :
* 29 # max_date = d1[month[:3]]
* )[(month[:3] == 'Feb') and leap_year] #
* if date <= max_date
* print ( '\n\n'.join(L1) )
* <_sre.SRE_Match object; span=(2, 35), match='MON Februar 12 0:30 : 19 CST 2018'>
* MON Februar 12 0:30 : 19 CST 2018
* {'day': 'MON', 'month': 'Februar', 'date': '12', 'hours': '0', 'minutes': '30', 'seconds': '19', 'time_zone': 'CST', 'year': '2018'}
* <_sre.SRE_Match object; span=(159, 255), match='Thursda feb 29 >
* Thursda feb 29 00:30:19 CST 1944
* {'day': 'Thursda', 'month': 'feb', 'date': '29', 'hours': '00', 'minutes': '30', 'seconds': '19', 'time_zone': 'CST', 'year': '1944'}

## Further Reading or Review

* Previous Lesson: With statement
* This Lesson:List Comprehension
* Next Lesson: Other statements
* Course Home Page