# Think Python/Lists

## A list is a sequence

Like a string, a **list** is a sequence of values. In a string, the
values are characters; in a list, they can be any type. The values in
list are called **elements** or sometimes **items**.

There are several ways to create a new list; the simplest is to enclose
the elements in square brackets (`[` and `]`):

``` python
[10, 20, 30, 40]
['crunchy frog', 'ram bladder', 'lark vomit']
```

The first example is a list of four integers. The second is a list of
three strings. The elements of a list don't have to be the same type.
The following list contains a string, a float, an integer, and (lo!)
another list:

``` python
['spam', 2.0, 5, [10, 20]]
```

A list within another list is **nested**.

A list that contains no elements is called an empty list; you can create
one with empty brackets, `[]`.

As you might expect, you can assign list values to variables:

``` python
>>> cheeses = ['Cheddar', 'Edam', 'Gouda']
>>> numbers = [17, 123]
>>> empty = []
>>> print cheeses, numbers, empty
['Cheddar', 'Edam', 'Gouda'] [17, 123] []
```

## Lists are mutable

The syntax for accessing the elements of a list is the same as for
accessing the characters of a string---the bracket operator. The
expression inside the brackets specifies the index. Remember that the
indices start at 0:

``` python
>>> print cheeses[0]
Cheddar
```

Unlike strings, lists are mutable. When the bracket operator appears on
the left side of an assignment, it identifies the element of the list
that will be assigned.

``` python
>>> numbers = [17, 123]
>>> numbers[1] = 5
>>> print numbers
[17, 5]
```

The one-eth element of `numbers`, which used to be 123, is now 5.

You can think of a list as a relationship between indices and elements.
This relationship is called a **mapping**; each index "maps to" one of
the elements. Here is a state diagram showing `cheeses`, `numbers` and
`empty`:

```{=html}
<DIV CLASS="center">
```
`<IMG SRC="book013.png">`{=html}

```{=html}
</DIV>
```
Lists are represented by boxes with the word "list" outside and the
elements of the list inside. `cheeses` refers to a list with three
elements indexed 0, 1 and 2. `numbers` contains two elements; the
diagram shows that the value of the second element has been reassigned
from 123 to 5. `empty` refers to a list with no elements.

List indices work the same way as string indices:

-   Any integer expression can be used as an index.
-   If you try to read or write an element that does not exist, you get
    an `IndexError`.
-   If an index has a negative value, it counts backward from the end of
    the list.

The `in` operator also works on lists.

``` python
>>> cheeses = ['Cheddar', 'Edam', 'Gouda']
>>> 'Edam' in cheeses
True
>>> 'Brie' in cheeses
False
```

## Traversing a list

The most common way to traverse the elements of a list is with a `for`
loop. The syntax is the same as for strings:

``` python
for cheese in cheeses:
    print cheese
```

This works well if you only need to read the elements of the list. But
if you want to write or update the elements, you need the indices. A
common way to do that is to combine the functions `range` and `len`:

``` python
for i in range(len(numbers)):
    numbers[i] = numbers[i] * 2
```

This loop traverses the list and updates each element. `len` returns the
number of elements in the list. `range` returns a list of indices from 0
to `<I>`{=html}n`</I>`{=html}−1, where `<I>`{=html}n`</I>`{=html} is the
length of the list. Each time through the loop `i` gets the index of the
next element. The assignment statement in the body uses `i` to read the
old value of the element and to assign the new value.

A `for` loop over an empty list never executes the body:

``` python
for x in empty:
    print 'This never happens.'
```

Although a list can contain another list, the nested list still counts
as a single element. The length of this list is four:

``` python
['spam', 1, ['Brie', 'Roquefort', 'Pol le Veq'], [1, 2, 3]]
```

## List operations

The `+` operator concatenates lists:

``` python
>>> a = [1, 2, 3]
>>> b = [4, 5, 6]
>>> c = a + b
>>> print c
[1, 2, 3, 4, 5, 6]
```

Similarly, the `*` operator repeats a list a given number of times:

``` python
>>> [0] * 4
[0, 0, 0, 0]
>>> [1, 2, 3] * 3
[1, 2, 3, 1, 2, 3, 1, 2, 3]
```

The first example repeats `[0]` four times. The second example repeats
the list `[1, 2, 3]` three times.

## List slices

The slice operator also works on lists:

``` python
>>> t = ['a', 'b', 'c', 'd', 'e', 'f']
>>> t[1:3]
['b', 'c']
>>> t[:4]
['a', 'b', 'c', 'd']
>>> t[3:]
['d', 'e', 'f']
```

If you omit the first index, the slice starts at the beginning. If you
omit the second, the slice goes to the end. So if you omit both, the
slice is a copy of the whole list.

``` python
>>> t[:]
['a', 'b', 'c', 'd', 'e', 'f']
```

Since lists are mutable, it is often useful to make a copy before
performing operations that fold, spindle or mutilate lists.

A slice operator on the left side of an assignment can update multiple
elements:

``` python
>>> t = ['a', 'b', 'c', 'd', 'e', 'f']
>>> t[1:3] = ['x', 'y']
>>> print t
['a', 'x', 'y', 'd', 'e', 'f']
```

## List methods

Python provides methods that operate on lists. For example, `append`
adds a new element to the end of a list:

``` python
>>> t = ['a', 'b', 'c']
>>> t.append('d')
>>> print t
['a', 'b', 'c', 'd']
```

`extend` takes a list as an argument and appends all of the elements:

``` python
>>> t1 = ['a', 'b', 'c']
>>> t2 = ['d', 'e']
>>> t1.extend(t2)
>>> print t1
['a', 'b', 'c', 'd', 'e']
```

This example leaves `t2` unmodified.

`sort` arranges the elements of the list from low to high:

``` python
>>> t = ['d', 'c', 'e', 'b', 'a']
>>> t.sort()
>>> print t
['a', 'b', 'c', 'd', 'e']
```

List methods are all void; they modify the list and return `None`. If
you accidentally write `t = t.sort()`, you will be disappointed with the
result.

## Map, filter and reduce

To add up all the numbers in a list, you can use a loop like this:

``` python
def add_all(t):
    total = 0
    for x in t:
        total += x
    return total
```

`total` is initialized to 0. Each time through the loop, `x` gets one
element from the list. The `+=` operator provides a short way to update
a variable:

``` python
    total += x
```

is equivalent to:

``` python
    total = total + x
```

As the loop executes, `total` accumulates the sum of the elements; a
variable used this way is sometimes called an **accumulator**.

Adding up the elements of a list is such a common operation that Python
provides it as a built-in function, `sum`:

``` python
>>> t = [1, 2, 3]
>>> sum(t)
6
```

An operation like this that combines a sequence of elements into a
single value is sometimes called **reduce**.

Sometimes you want to traverse one list while building another. For
example, the following function takes a list of strings and returns a
new list that contains capitalized strings:

``` python
def capitalize_all(t):
    res = []
    for s in t:
        res.append(s.capitalize())
    return res
```

`res` is initialized with an empty list; each time through the loop, we
append the next element. So `res` is another kind of accumulator.

An operation like `capitalize_all` is sometimes called a **map** because
it "maps" a function (in this case the method `capitalize`) onto each of
the elements in a sequence.

Another common operation is to select some of the elements from a list
and return a sublist. For example, the following function takes a list
of strings and returns a list that contains only the uppercase strings:

``` python
def only_upper(t):
    res = []
    for s in t:
        if s.isupper():
            res.append(s)
    return res
```

`isupper` is a string method that returns `True` if the string contains
only upper case letters.

An operation like `only_upper` is called a **filter** because it selects
some of the elements and filters out the others.

Most common list operations can be expressed as a combination of map,
filter and reduce. Because these operations are so common, Python
provides language features to support them, including the built-in
function `map` and an operator called a "list comprehension."

### Exercise 1

\'\'Write a function that takes a list of numbers and returns the
cumulative sum; that is, a new list where the
**\'`<I>`{=html}i`</I>`{=html}**\'th element is the sum of the first
**\'`<I>`{=html}i`</I>`{=html}+1**\' elements from the original list.
For example, the cumulative sum of **\'`[1, 2, 3]`**\' is
**\'`[1, 3, 6]`**\'.

## Deleting elements

There are several ways to delete elements from a list. If you know the
index of the element you want, you can use `pop`:

``` python
>>> t = ['a', 'b', 'c']
>>> x = t.pop(1)
>>> print t
['a', 'c']
>>> print x
b
```

`pop` modifies the list and returns the element that was removed. If you
don't provide an index, it deletes and returns the last element.

If you don't need the removed value, you can use the `del` operator:

``` python
>>> t = ['a', 'b', 'c']
>>> del t[1]
>>> print t
['a', 'c']
```

If you know the element you want to remove (but not the index), you can
use `remove`:

``` python
>>> t = ['a', 'b', 'c']
>>> t.remove('b')
>>> print t
['a', 'c']
```

The return value from `remove` is `None`.

To remove more than one element, you can use `del` with a slice index:

``` python
>>> t = ['a', 'b', 'c', 'd', 'e', 'f']
>>> del t[1:5]
>>> print t
['a', 'f']
```

As usual, the slice selects all the elements up to, but not including,
the second index.

## Lists and strings

A string is a sequence of characters and a list is a sequence of values,
but a list of characters is not the same as a string. To convert from a
string to a list of characters, you can use `list`:

``` python
>>> s = 'spam'
>>> t = list(s)
>>> print t
['s', 'p', 'a', 'm']
```

Because `list` is the name of a built-in function, you should avoid
using it as a variable name. I also avoid `l` because it looks too much
like `1`. So that's why I use `t`.

The `list` function breaks a string into individual letters. If you want
to break a string into words, you can use the `split` method:

``` python
>>> s = 'pining for the fjords'
>>> t = s.split()
>>> print t
['pining', 'for', 'the', 'fjords']
```

An optional argument called a **delimiter** specifies which characters
to use as word boundaries. The following example uses a hyphen as a
delimiter:

``` python
>>> s = 'spam-spam-spam'
>>> delimiter = '-'
>>> s.split(delimiter)
['spam', 'spam', 'spam']
```

`join` is the inverse of `split`. It takes a list of strings and
concatenates the elements. `join` is a string method, so you have to
invoke it on the delimiter and pass the list as a parameter:

``` python
>>> t = ['pining', 'for', 'the', 'fjords']
>>> delimiter = ' '
>>> delimiter.join(t)
'pining for the fjords'
```

In this case the delimiter is a space character, so `join` puts a space
between words. To concatenate strings without spaces, you can use the
empty string, `''`, as a delimiter.

## Objects and values

If we execute these assignment statements:

``` python
a = 'banana'
b = 'banana'
```

We know that `a` and `b` both refer to a string, but we don't know
whether they refer to the *same* string. There are two possible states:

```{=html}
<DIV CLASS="center">
```
`<IMG SRC="book014.png">`{=html}

```{=html}
</DIV>
```
In one case, `a` and `b` refer to two different objects that have the
same value. In the second case, they refer to the same object.

To check whether two variables refer to the same object, you can use the
`is` operator.

``` python
>>> a = 'banana'
>>> b = 'banana'
>>> a is b
True
```

In this example, Python only created one string object, and both `a` and
`b` refer to it.

But when you create two lists, you get two objects:

``` python
>>> a = [1, 2, 3]
>>> b = [1, 2, 3]
>>> a is b
False
```

So the state diagram looks like this:

```{=html}
<DIV CLASS="center">
```
`<IMG SRC="book015.png">`{=html}

```{=html}
</DIV>
```
In this case we would say that the two lists are **equivalent**, because
they have the same elements, but not **identical**, because they are not
the same object. If two objects are identical, they are also equivalent,
but if they are equivalent, they are not necessarily identical.

Until now, we have been using "object" and "value" interchangeably, but
it is more precise to say that an object has a value. If you execute
`a = [1,2,3]`, `a` refers to a list object whose value is a particular
sequence of elements. If another list has the same elements, we would
say it has the same value.

## Aliasing

If `a` refers to an object and you assign `b = a`, then both variables
refer to the same object:

``` python
>>> a = [1, 2, 3]
>>> b = a
>>> b is a
True
```

The state diagram looks like this:

```{=html}
<DIV CLASS="center">
```
`<IMG SRC="book016.png">`{=html}

```{=html}
</DIV>
```
The association of a variable with an object is called a **reference**.
In this example, there are two references to the same object.

An object with more than one reference has more than one name, so we say
that the object is **aliased**.

If the aliased object is mutable, changes made with one alias affect the
other:

``` python
>>> b[0] = 17
>>> print a
[17, 2, 3]
```

Although this behavior can be useful, it is error-prone. In general, it
is safer to avoid aliasing when you are working with mutable objects.

For immutable objects like strings, aliasing is not as much of a
problem. In this example:

``` python
a = 'banana'
b = 'banana'
```

It almost never makes a difference whether `a` and `b` refer to the same
string or not.

## List arguments

When you pass a list to a function, the function gets a reference to the
list. If the function modifies a list parameter, the caller sees the
change. For example, `delete_head` removes the first element from a
list:

``` python
def delete_head(t):
    del t[0]
```

Here's how it is used:

``` python
>>> letters = ['a', 'b', 'c']
>>> delete_head(letters)
>>> print letters
['b', 'c']
```

The parameter `t` and the variable `letters` are aliases for the same
object. The stack diagram looks like this:

```{=html}
<DIV CLASS="center">
```
`<IMG SRC="book017.png">`{=html}

```{=html}
</DIV>
```
Since the list is shared by two frames, I drew it between them.

It is important to distinguish between operations that modify lists and
operations that create new lists. For example, the `append` method
modifies a list, but the `+` operator creates a new list:

``` python
>>> t1 = [1, 2]
>>> t2 = t1.append(3)
>>> print t1
[1, 2, 3]
>>> print t2
None

>>> t3 = t1 + [3]
>>> print t3
[1, 2, 3]
>>> t2 is t3
False
```

This difference is important when you write functions that are supposed
to modify lists. For example, this function *does not* delete the head
of a list:

``` python
def bad_delete_head(t):
    t = t[1:]              # WRONG!
```

The slice operator creates a new list and the assignment makes `t` refer
to it, but none of that has any effect on the list that was passed as an
argument.

An alternative is to write a function that creates and returns a new
list. For example, `tail` returns all but the first element of a list:

``` python
def tail(t):
    return t[1:]
```

This function leaves the original list unmodified. Here's how it is
used:

``` python
>>> letters = ['a', 'b', 'c']
>>> rest = tail(letters)
>>> print rest
['b', 'c']
```

### Exercise 2

Write a function called **\'`chop`**\' that takes a list and modifies
it, removing the first and last elements, and returns **\'`None`**\'.

Then write a function called **\'`middle`**\' that takes a list and
returns a new list that contains all but the first and last elements.

## Debugging

Careless use of lists (and other mutable objects) can lead to long hours
of debugging. Here are some common pitfalls and ways to avoid them:

-   Don't forget that most list methods modify the argument and

return `None`. This is the opposite of the string methods, which return
a new string and leave the original alone. If you are used to writing
string code like this:

``` python
word = word.strip()
```

It is tempting to write list code like this:

``` python
t = t.sort()           # WRONG!
```

Because `sort` returns `None`, the next operation you perform with `t`
is likely to fail.

Before using list methods and operators, you should read the
documentation carefully and then test them in interactive mode. The
methods and operators that lists share with other sequences (like
strings) are documented at `docs.python.org/lib/typesseq.html`. The
methods and operators that only apply to mutable sequences are
documented at `docs.python.org/lib/typesseq-mutable.html`.

-   Pick an idiom and stick with it.

Part of the problem with lists is that there are too many ways to do
things. For example, to remove an element from a list, you can use
`pop`, `remove`, `del`, or even a slice assignment.

To add an element, you can use the `append` method or the `+` operator.
But don't forget that these are right:

``` python
t.append(x)
t = t + [x]
```

And these are wrong:

``` python
t.append([x])          # WRONG!
t = t.append(x)        # WRONG!
t + [x]                # WRONG!
t = t + x              # WRONG!
```

Try out each of these examples in interactive mode to make sure you
understand what they do. Notice that only the last one causes a runtime
error; the other three are legal, but they do the wrong thing.

-   Make copies to avoid aliasing.

If you want to use a method like `sort` that modifies the argument, but
you need to keep the original list as well, you can make a copy.

``` python
orig = t[:]
t.sort()
```

In this example you could also use the built-in function `sorted`, which
returns a new, sorted list and leaves the original alone. But in that
case you should avoid using `sorted` as a variable name!

## Glossary

```{=html}
<DL CLASS="description">
```
```{=html}
<DT CLASS="dt-description">
```
**list:**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
A sequence of values.

```{=html}
</DD>
```
```{=html}
<DT CLASS="dt-description">
```
**element:**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
One of the values in a list (or other sequence), also called items.

```{=html}
</DD>
```
```{=html}
<DT CLASS="dt-description">
```
**index:**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
An integer value that indicates an element in a list.

```{=html}
</DD>
```
```{=html}
<DT CLASS="dt-description">
```
**nested list:**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
A list that is an element of another list.

```{=html}
</DD>
```
```{=html}
<DT CLASS="dt-description">
```
**list traversal:**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
The sequential accessing of each element in a list.

```{=html}
</DD>
```
```{=html}
<DT CLASS="dt-description">
```
**mapping:**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
A relationship in which each element of one set corresponds to an
element of another set. For example, a list is a mapping from indices to
elements.

```{=html}
</DD>
```
```{=html}
<DT CLASS="dt-description">
```
**accumulator:**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
A variable used in a loop to add up or accumulate a result.

```{=html}
</DD>
```
```{=html}
<DT CLASS="dt-description">
```
**reduce:**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
A processing pattern that traverses a sequence and accumulates the
elements into a single result.

```{=html}
</DD>
```
```{=html}
<DT CLASS="dt-description">
```
**map:**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
A processing pattern that traverses a sequence and performs an operation
on each element.

```{=html}
</DD>
```
```{=html}
<DT CLASS="dt-description">
```
**filter:**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
A processing pattern that traverses a list and selects the elements that
satisfy some criterion.

```{=html}
</DD>
```
```{=html}
<DT CLASS="dt-description">
```
**object:**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
Something a variable can refer to. An object has a type and a value.

```{=html}
</DD>
```
```{=html}
<DT CLASS="dt-description">
```
**equivalent:**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
Having the same value.

```{=html}
</DD>
```
```{=html}
<DT CLASS="dt-description">
```
**identical:**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
Being the same object (which implies equivalence).

```{=html}
</DD>
```
```{=html}
<DT CLASS="dt-description">
```
**reference:**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
The association between a variable and its value.

```{=html}
</DD>
```
```{=html}
<DT CLASS="dt-description">
```
**aliasing:**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
A circumstance where two variables refer to the same object.

```{=html}
</DD>
```
```{=html}
<DT CLASS="dt-description">
```
**delimiter:**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
A character or string used to indicate where a string should be split.

```{=html}
</DD>
```
```{=html}
</DL>
```
## Exercises

### Exercise 3

Write a function called **`is_sorted`** that takes a list as a parameter
and returns **\'`True`**\' if the list is sorted in ascending order and
**\'`False`**\' otherwise. You can assume (as a precondition) that the
elements of the list can be compared with the comparison operators
**\'`<`**\', **\'`>`**\', etc.\'\'

*For example,`is_sorted([1,2,2])`should return **\'`True`**\'
and`is_sorted(['b','a'])`*\'\' should return **\'`False`**\'.

### Exercise 4

*Two words are anagrams if you can rearrange the letters from one to
spell the other. Write a function called`is_anagram`that takes two
strings and returns **\'`True`**\' if they are anagrams.*

```{=html}
</DIV>
```
```{=html}
<DIV CLASS="theorem">
```
**Exercise 5**   *The (so-called) Birthday Paradox:*

-

*Write a function called`has_duplicates`that takes a list and returns
**\'`True`**\' if there is any element that appears more than once. It
should not modify the original list.*

-   \'\'If there are 23 students in your class, what are the chances

that two of you have the same birthday? You can estimate this
probability by generating random samples of 23 birthdays and checking
for matches. Hint: you can generate random birthdays with the
**\'`randint`**\' function in the **\'`random`**\' module.\'\'

*You can read about this problem at
**\'`wikipedia.org/wiki/Birthday_paradox`**\', and you can see my
solution at **\'`thinkpython.com/code/birthday.py`**\'.*

```{=html}
</DIV>
```
```{=html}
<DIV CLASS="theorem">
```
**Exercise 6**

*Write a function called`remove_duplicates`*\'\' that takes a list and
returns a new list with only the unique elements from the original.
Hint: they don't have to be in the same order.

### Exercise 7

*Write a function that reads the file **\'`words.txt`**\' and builds a
list with one element per word. Write two versions of this function, one
using the **\'`append`**\' method and the other using the idiom
**\'`t = t + [x]`**\'. Which one takes longer to run? Why?*

\'\'You can see my solution at
**\'`thinkpython.com/code/wordlist.py`**\'.

### Exercise 8

*To check whether a word is in the word list, you could use the
**\'`in`**\' operator, but it would be slow because it searches through
the words in order.*

*Because the words are in alphabetical order, we can speed things up
with a bisection search, which is similar to what you do when you look a
word up in the dictionary. You start in the middle and check to see
whether the word you are looking for comes before the word in the middle
of the list. If so, then you search the first half of the list the same
way. Otherwise you search the second half.*

*Either way, you cut the remaining search space in half. If the word
list has 113,809 words, it will take about 17 steps to find the word or
conclude that it's not there.*

*Write a function called **\'`bisect`**\' that takes a sorted list and a
target value and returns the index of the value in the list, if it's
there, or **\'`None`**\' if it's not.*

*Or you could read the documentation of the **\'`bisect`**\' module and
use that!*

### Exercise 9

Two words are a "reverse pair" if each is the reverse of the other.
Write a program that finds all the reverse pairs in the word list.

### Exercise 10

*Two words "interlock" if taking alternating letters from each forms a
new word*^*1*^*. For example, "shoe" and "cold" interlock to form
"schooled."*

-   \'\'Write a program that finds all pairs of words that interlock.

Hint: don't enumerate all pairs!\'\'

-   \'\'Can you find any words that are three-way interlocked; that is,

every third letter forms a word, starting from the first, second or
third?\'\'

```{=html}
</DIV>
```
```{=html}
<HR CLASS="footnoterule">
```
```{=html}
<DL CLASS="thefootnotes">
```
```{=html}
<DT CLASS="dt-thefootnotes">
```
1

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-thefootnotes">
```
This exercise is inspired by an example at `puzzlers.org`.

```{=html}
</DD>
```
```{=html}
</DL>
```


# Think Python/Dictionaries

A **dictionary** is like a list, but more general. In a list, the
indices have to be integers; in a dictionary they can be (almost) any
type.

You can think of a dictionary as a mapping between a set of indices
(which are called **keys**) and a set of values. Each key maps to a
value. The association of a key and a value is called a **key-value
pair** or sometimes an **item**.

As an example, we\'ll build a dictionary that maps from English to
Spanish words, so the keys and the values are all strings.

The function `dict` creates a new dictionary with no items. Because
`dict` is the name of a built-in function, you should avoid using it as
a variable name.

``` python
>>> eng2sp = dict()
>>> print eng2sp
{}
```

The squiggly-brackets, `{}`, represent an empty dictionary. To add items
to the dictionary, you can use square brackets:

``` python
>>> eng2sp['one'] = 'uno'
```

This line creates an item that maps from the key `’one’` to the value
`'uno'`. If we print the dictionary again, we see a key-value pair with
a colon between the key and value:

``` python
>>> print eng2sp
{'one': 'uno'}
```

This output format is also an input format. For example, you can create
a new dictionary with three items:

``` python
>>> eng2sp = {'one': 'uno', 'two': 'dos', 'three': 'tres'}
```

But if you print `eng2sp`, you might be surprised:

``` python
>>> print eng2sp
{'one': 'uno', 'three': 'tres', 'two': 'dos'}
```

The order of the key-value pairs is not the same. In fact, if you type
the same example on your computer, you might get a different result. In
general, the order of items in a dictionary is unpredictable.

But that's not a problem because the elements of a dictionary are never
indexed with integer indices. Instead, you use the keys to look up the
corresponding values:

``` python
>>> print eng2sp['two']
'dos'
```

The key `’two’` always maps to the value `'dos'` so the order of the
items doesn't matter.

If the key isn't in the dictionary, you get an exception:

``` python
>>> print eng2sp['four']
KeyError: 'four'
```

The `len` function works on dictionaries; it returns the number of
key-value pairs:

``` python
>>> len(eng2sp)
3
```

The `in` operator works on dictionaries; it tells you whether something
appears as a *key* in the dictionary (appearing as a value is not good
enough).

``` python
>>> 'one' in eng2sp
True
>>> 'uno' in eng2sp
False
```

To see whether something appears as a value in a dictionary, you can use
the method `values`, which returns the values as a list, and then use
the `in` operator:

``` python
>>> vals = eng2sp.values()
>>> 'uno' in vals
True
```

You can do the same with `keys`:

``` python
>>> kees = eng2sp.keys()
>>> ‘one’ in kees
True
```

The `in` operator uses different algorithms for lists and dictionaries.
For lists, it uses a search algorithm, as in Section 8.6. As the list
gets longer, the search time gets longer in direct proportion. For
dictionaries, Python uses an algorithm called a **hashtable** that has a
remarkable property: the `in` operator takes about the same amount of
time no matter how many items there are in a dictionary. I won't explain
how that's possible, but you can read more about it at
`wikipedia.org/wiki/Hash_table`.

### Exercise 1

*Write a function that reads the words in **\'`words.txt`**\' and stores
them as keys in a dictionary. It doesn't matter what the values are.
Then you can use the **\'`in`**\' operator as a fast way to check
whether a string is in the dictionary.*

*If you did Exercise **\'10.8**\', you can compare the speed of this
implementation with the list **\'`in`**\' operator and the bisection
search.*

```{=html}
</DIV>
```
## Dictionary as a set of counters

Suppose you are given a string and you want to count how many times each
letter appears. There are several ways you could do it:

-   You could create 26 variables, one for each letter of the alphabet.
    Then you could traverse the string and, for each character,
    increment the corresponding counter, probably using a chained
    conditional.
-   You could create a list with 26 elements. Then you could convert
    each character to a number (using the built-in function `ord`), use
    the number as an index into the list, and increment the appropriate
    counter.
-   You could create a dictionary with characters as keys and counters
    as the corresponding values. The first time you see a character, you
    would add an item to the dictionary. After that you would increment
    the value of an existing item.

Each of these options performs the same computation, but each of them
implements that computation in a different way.

An **implementation** is a way of performing a computation; some
implementations are better than others. For example, an advantage of the
dictionary implementation is that we don't have to know ahead of time
which letters appear in the string and we only have to make room for the
letters that do appear.

Here is what the code might look like:

``` python
def histogram(s):
    d = dict()
    for c in s:
        if c not in d:
            d[c] = 1
        else:
            d[c] += 1
    return d
```

The name of the function is **histogram**, which is a statistical term
for a set of counters (or frequencies).

The first line of the function creates an empty dictionary. The `for`
loop traverses the string. Each time through the loop, if the character
`c` is not in the dictionary, we create a new item with key `c` and the
initial value 1 (since we have seen this letter once). If `c` is already
in the dictionary we increment `d[c]`.

Here's how it works:

``` python
>>> h = histogram('brontosaurus')
>>> print h
{'a': 1, 'b': 1, 'o': 2, 'n': 1, 's': 2, 'r': 2, 'u': 2, 't': 1}
```

The histogram indicates that the letters `’a’` and `'b'` appear once;
`'o'` appears twice, and so on.

### Exercise 2

*Dictionaries have a method called **\'`get`**\' that takes a key and a
default value. If the key appears in the dictionary, **\'`get`**\'
returns the corresponding value; otherwise it returns the default value.
For example:*

``` python
''>>> h = histogram('a')
>>> print h
{'a': 1}
>>> h.get('a', 0)
1
>>> h.get('b', 0)
0
''
```

*Use **\'`get`**\' to write **\'`histogram`**\' more concisely. You
should be able to eliminate the **\'`if`**\' statement.*

```{=html}
</DIV>
```
## Looping and dictionaries

If you use a dictionary in a `for` statement, it traverses the keys of
the dictionary. For example, `print_hist` prints each key and the
corresponding value:

``` python
def print_hist(h):
    for c in h:
        print c, h[c]
```

Here's what the output looks like:

``` python
>>> h = histogram('parrot')
>>> print_hist(h)
a 1
p 1
r 2
t 1
o 1
```

Again, the keys are in no particular order.

### Exercise 3

*Dictionaries have a method called **\'`keys`**\' that returns the keys
of the dictionary, in no particular order, as a list.*

*Modify`print_hist`*\'\' to print the keys and their values in
alphabetical order.

## Reverse lookup

Given a dictionary `d` and a key `k`, it is easy to find the
corresponding value `v = d[k]`. This operation is called a **lookup**.

But what if you have `v` and you want to find `k`? You have two
problems: first, there might be more than one key that maps to the value
`v`. Depending on the application, you might be able to pick one, or you
might have to make a list that contains all of them. Second, there is no
simple syntax to do a **reverse lookup**; you have to search.

Here is a function that takes a value and returns the first key that
maps to that value:

``` python
def reverse_lookup(d, v):
    for k in d:
        if d[k] == v:
            return k
    raise ValueError
```

This function is yet another example of the search pattern, but it uses
a feature we haven't seen before, `raise`. The `raise` statement causes
an exception; in this case it causes a `ValueError`, which generally
indicates that there is something wrong with the value of a parameter.

If we get to the end of the loop, that means `v` doesn't appear in the
dictionary as a value, so we raise an exception.

Here is an example of a successful reverse lookup:

``` python
>>> h = histogram('parrot')
>>> k = reverse_lookup(h, 2)
>>> print k
r
```

And an unsuccessful one:

``` python
>>> k = reverse_lookup(h, 3)
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "<stdin>", line 5, in reverse_lookup
ValueError
```

The result when you raise an exception is the same as when Python raises
one: it prints a traceback and an error message.

The `raise` statement takes a detailed error message as an optional
argument. For example:

``` python
>>> raise ValueError, 'value does not appear in the dictionary'
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
ValueError: value does not appear in the dictionary
```

A reverse lookup is much slower than a forward lookup; if you have to do
it often, or if the dictionary gets big, the performance of your program
will suffer.

```{=html}
<DIV CLASS="theorem">
```
**Exercise 4**  *Modify`reverse_lookup`so that it builds and returns a
list of*all*keys that map to **\'`v`**\', or an empty list if there are
none.*

```{=html}
</DIV>
```
## Dictionaries and lists

Lists can appear as values in a dictionary. For example, if you were
given a dictionary that maps from letters to frequencies, you might want
to invert it; that is, create a dictionary that maps from frequencies to
letters. Since there might be several letters with the same frequency,
each value in the inverted dictionary should be a list of letters.

Here is a function that inverts a dictionary:

``` python
def invert_dict(d):
    inv = dict()
    for key in d:
        val = d[key]
        if val not in inv:
            inv[val] = [key]
        else:
            inv[val].append(key)
    return inv
```

Each time through the loop, `key` gets a key from `d` and `val` gets the
corresponding value. If `val` is not in `inv`, that means we haven't
seen it before, so we create a new item and initialize it with a
**singleton** (a list that contains a single element). Otherwise we have
seen this value before, so we append the corresponding key to the list.

Here is an example:

``` python
>>> hist = histogram('parrot')
>>> print hist
{'a': 1, 'p': 1, 'r': 2, 't': 1, 'o': 1}
>>> inv = invert_dict(hist)
>>> print inv
{1: ['a', 'p', 't', 'o'], 2: ['r']}
```

And here is a diagram showing `hist` and `inv`:

```{=html}
<DIV CLASS="center">
```
`<IMG SRC="book018.png">`{=html}

```{=html}
</DIV>
```
A dictionary is represented as a box with the type `dict` above it and
the key-value pairs inside. If the values are integers, floats or
strings, I usually draw them inside the box, but I usually draw lists
outside the box, just to keep the diagram simple.

Lists can be values in a dictionary, as this example shows, but they
cannot be keys. Here's what happens if you try:

``` python
>>> t = [1, 2, 3]
>>> d = dict()
>>> d[t] = 'oops'
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
TypeError: list objects are unhashable
```

I mentioned earlier that a dictionary is implemented using a hashtable
and that means that the keys have to be **hashable**.

A **hash** is a function that takes a value (of any kind) and returns an
integer. Dictionaries use these integers, called hash values, to store
and look up key-value pairs.

This system works fine if the keys are immutable. But if the keys are
mutable, like lists, bad things happen. For example, when you create a
key-value pair, Python hashes the key and stores it in the corresponding
location. If you modify the key and then hash it again, it would go to a
different location. In that case you might have two entries for the same
key, or you might not be able to find a key. Either way, the dictionary
wouldn't work correctly.

That's why the keys have to be hashable, and why mutable types like
lists aren't. The simplest way to get around this limitation is to use
tuples, which we will see in the next chapter.

Since dictionaries are mutable, they can't be used as keys, but they
*can* be used as values.

### Exercise 5

Read the documentation of the dictionary method **\'`setdefault`**\' and
use it to write a more concise version of **`invert_dict`**.\'\'

## Memos

If you played with the `fibonacci` function from Section 6.7, you might
have noticed that the bigger the argument you provide, the longer the
function takes to run. Furthermore, the run time increases very quickly.

To understand why, consider this **call graph** for `fibonacci` with
`n=4`:

```{=html}
<DIV CLASS="center">
```
`<IMG SRC="book019.png">`{=html}

```{=html}
</DIV>
```
A call graph shows a set of function frames, with lines connecting each
frame to the frames of the functions it calls. At the top of the graph,
`fibonacci` with `n=4` calls `fibonacci` with `n=3` and `n=2`. In turn,
`fibonacci` with `n=3` calls `fibonacci` with `n=2` and `n=1`. And so
on.

Count how many times `fibonacci(0)` and `fibonacci(1)` are called. This
is an inefficient solution to the problem, and it gets worse as the
argument gets bigger.

One solution is to keep track of values that have already been computed
by storing them in a dictionary. A previously computed value that is
stored for later use is called a **memo**[^1]. Here is an implementation
of `fibonacci` using memos:

``` python
known = {0:0, 1:1}

def fibonacci(n):
    if n in known:
        return known[n]

    res = fibonacci(n-1) + fibonacci(n-2)
    known[n] = res
    return res
```

`known` is a dictionary that keeps track of the Fibonacci numbers we
already know. It starts with two items: 0 maps to 0 and 1 maps to 1.

Whenever `fibonacci` is called, it checks `known`. If the result is
already there, it can return immediately. Otherwise it has to compute
the new value, add it to the dictionary, and return it.

### Exercise 6

Run this version of **\'`fibonacci`**\' and the original with a range of
parameters and compare their run times.

## Global variables

In the previous example, `known` is created outside the function, so it
belongs to the special frame called `__main__`. Variables in `__main__`
are sometimes called **global** because they can be accessed from any
function. Unlike local variables, which disappear when their function
ends, global variables persist from one function call to the next.

It is common to use global variables for **flags**; that is, boolean
variables that indicate ("flag") whether a condition is true. For
example, some programs use a flag named `verbose` to control the level
of detail in the output:

``` python
verbose = True

def example1():
    if verbose:
        print 'Running example1'
```

If you try to reassign a global variable, you might be surprised. The
following example is supposed to keep track of whether the function has
been called:

``` python
been_called = False

def example2():
    been_called = True         # WRONG
```

But if you run it you will see that the value of `been_called` doesn't
change. The problem is that `example2` creates a new local variable
named `been_called`. The local variable goes away when the function
ends, and has no effect on the global variable.

To reassign a global variable inside a function you have to **declare**
the global variable before you use it:

``` python
been_called = False

def example2():
    global been_called
    been_called = True
```

The `global` statement tells the interpreter something like, "In this
function, when I say `been_called`, I mean the global variable; don't
create a local one."

Here's an example that tries to update a global variable:

``` python
count = 0

def example3():
    count = count + 1          # WRONG
```

If you run it you get:

``` python
UnboundLocalError: local variable 'count' referenced before assignment
```

Python assumes that `count` is local, which means that you are reading
it before writing it. The solution, again, is to declare `count` global.

``` python
def example3():
    global count
    count += 1
```

If the global value is mutable, you can modify it without declaring it:

``` python
known = {0:0, 1:1}

def example4():
    known[2] = 1
```

So you can add, remove and replace elements of a global list or
dictionary, but if you want to reassign the variable, you have to
declare it:

``` python
def example5():
    global known
    known = dict()
```

## Long integers

If you compute `fibonacci(50)`, you get:

``` python
>>> fibonacci(50)
12586269025L
```

The `L` at the end indicates that the result is a long integer[^2], or
type `long`.

Values with type `int` have a limited range; long integers can be
arbitrarily big, but as they get bigger they consume more space and
time.

The mathematical operators work on long integers, and the functions in
the `math` module, too, so in general any code that works with `int`
will also work with `long`.

Any time the result of a computation is too big to be represented with
an integer, Python converts the result as a long integer:

``` python
>>> 1000 * 1000
1000000
>>> 100000 * 100000
10000000000L
```

In the first case the result has type `int`; in the second case it is
`long`.

### Exercise 7

*Exponentiation of large integers is the basis of common algorithms for
public-key encryption. Read the Wikipedia page on the RSA algorithm[^3]
and write functions to encode and decode messages.*

## Debugging

As you work with bigger datasets it can become unwieldy to debug by
printing and checking data by hand. Here are some suggestions for
debugging large datasets:

```{=html}
<DL CLASS="description">
```
```{=html}
<DT CLASS="dt-description">
```
**Scale down the input:**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
If possible, reduce the size of the dataset. For example if the program
reads a text file, start with just the first 10 lines, or with the
smallest example you can find. You can either edit the files themselves,
or (better) modify the program so it reads only the first `n` lines. If
there is an error, you can reduce `n` to the smallest value that
manifests the error, and then increase it gradually as you find and
correct errors.

```{=html}
</DD>
```
```{=html}
<DT CLASS="dt-description">
```
**Check summaries and types:**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
Instead of printing and checking the entire dataset, consider printing
summaries of the data: for example, the number of items in a dictionary
or the total of a list of numbers. A common cause of runtime errors is a
value that is not the right type. For debugging this kind of error, it
is often enough to print the type of a value.

```{=html}
</DD>
```
```{=html}
<DT CLASS="dt-description">
```
**Write self-checks:**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
Sometimes you can write code to check for errors automatically. For
example, if you are computing the average of a list of numbers, you
could check that the result is not greater than the largest element in
the list or less than the smallest. This is called a "sanity check"
because it detects results that are "insane."

Another kind of check compares the results of two different computations
to see if they are consistent. This is called a "consistency check."

```{=html}
</DD>
```
```{=html}
<DT CLASS="dt-description">
```
**Pretty print the output:**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
Formatting debugging output can make it easier to spot an error. We saw
an example in Section 6.9. The `pprint` module provides a `pprint`
function that displays built-in types in a more human-readable format.

```{=html}
</DD>
```
```{=html}
</DL>
```
Again, time you spend building scaffolding can reduce the time you spend
debugging.

## Glossary

```{=html}
<DL CLASS="description">
```
```{=html}
<DT CLASS="dt-description">
```
**dictionary:**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
A mapping from a set of keys to their corresponding values.

```{=html}
</DD>
```
```{=html}
<DT CLASS="dt-description">
```
**key-value pair:**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
The representation of the mapping from a key to a value.

```{=html}
</DD>
```
```{=html}
<DT CLASS="dt-description">
```
**item:**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
Another name for a key-value pair.

```{=html}
</DD>
```
```{=html}
<DT CLASS="dt-description">
```
**key:**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
An object that appears in a dictionary as the first part of a key-value
pair.

```{=html}
</DD>
```
```{=html}
<DT CLASS="dt-description">
```
**value:**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
An object that appears in a dictionary as the second part of a key-value
pair. This is more specific than our previous use of the word "value."

```{=html}
</DD>
```
```{=html}
<DT CLASS="dt-description">
```
**implementation:**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
A way of performing a computation.

```{=html}
</DD>
```
```{=html}
<DT CLASS="dt-description">
```
**hashtable:**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
The algorithm used to implement Python dictionaries.

```{=html}
</DD>
```
```{=html}
<DT CLASS="dt-description">
```
**hash function:**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
A function used by a hashtable to compute the location for a key.

```{=html}
</DD>
```
```{=html}
<DT CLASS="dt-description">
```
**hashable:**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
A type that has a hash function. Immutable types like integers, floats
and strings are hashable; mutable types like lists and dictionaries are
not.

```{=html}
</DD>
```
```{=html}
<DT CLASS="dt-description">
```
**lookup:**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
A dictionary operation that takes a key and finds the corresponding
value.

```{=html}
</DD>
```
```{=html}
<DT CLASS="dt-description">
```
**reverse lookup:**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
A dictionary operation that takes a value and finds one or more keys
that map to it.

```{=html}
</DD>
```
```{=html}
<DT CLASS="dt-description">
```
**singleton:**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
A list (or other sequence) with a single element.

```{=html}
</DD>
```
```{=html}
<DT CLASS="dt-description">
```
**call graph:**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
A diagram that shows every frame created during the execution of a
program, with an arrow from each caller to each callee.

```{=html}
</DD>
```
```{=html}
<DT CLASS="dt-description">
```
**histogram:**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
A set of counters.

```{=html}
</DD>
```
```{=html}
<DT CLASS="dt-description">
```
**memo:**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
A computed value stored to avoid unnecessary future computation.

```{=html}
</DD>
```
```{=html}
<DT CLASS="dt-description">
```
**global variable:**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
A variable defined outside a function. Global variables can be accessed
from any function.

```{=html}
</DD>
```
```{=html}
<DT CLASS="dt-description">
```
**flag:**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
A boolean variable used to indicate whether a condition is true.

```{=html}
</DD>
```
```{=html}
<DT CLASS="dt-description">
```
**declaration:**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
A statement like `global` that tells the interpreter something about a
variable.

```{=html}
</DD>
```
```{=html}
</DL>
```
## Exercise-8

`        Dictionaries have a method called 'keys' that returns the keys of the dictionary, in`

no particular order, as a list. Modify print_hist to print the keys and
their values in alphabetical order.

### Exercise 9

*Two words are "rotate pairs" if you can rotate one of them and get the
other (see`rotate_word`in Exercise **\'8.12**\').*

\'\'Write a program that reads a wordlist and finds all the rotate
pairs.

### Exercise 10

*Here's another Puzzler from*Car Talk[^4]*:*

> *This was sent in by a fellow named Dan O'Leary. He came upon a common
> one-syllable, five-letter word recently that has the following unique
> property. When you remove the first letter, the remaining letters form
> a homophone of the original word, that is a word that sounds exactly
> the same. Replace the first letter, that is, put it back and remove
> the second letter and the result is yet another homophone of the
> original word. And the question is, what's the word?* *Now I'm going
> to give you an example that doesn't work. Let's look at the
> five-letter word, 'wrack.' W-R-A-C-K, you know like to 'wrack with
> pain.' If I remove the first letter, I am left with a four-letter
> word, 'R-A-C-K.' As in, 'Holy cow, did you see the rack on that buck!
> It must have been a nine-pointer!' It's a perfect homophone. If you
> put the 'w' back, and remove the 'r,' instead, you're left with the
> word, 'wack,' which is a real word, it's just not a homophone of the
> other two words.*
>
> *But there is, however, at least one word that Dan and we know of,
> which will yield two homophones if you remove either of the first two
> letters to make two, new four-letter words. The question is, what's
> the word?*

*You can use the dictionary from Exercise **\'11.1**\' to check whether
a string is in the word list.*

*To check whether two words are homophones, you can use the CMU
Pronouncing Dictionary. You can download it from
**\'`www.speech.cs.cmu.edu/cgi-bin/cmudict`**\' or from
**\'`thinkpython.com/code/c06d`**\' and you can also download
**\'`thinkpython.com/code/pronounce.py`**\', which provides a function
named`read_dictionary`that reads the pronouncing dictionary and returns
a Python dictionary that maps from each word to a string that describes
its primary pronunciation.*

*Write a program that lists all the words that solve the Puzzler. You
can see my solution at **\'`thinkpython.com/code/homophone.py`**\'.*

## Notes

[^1]: See `wikipedia.org/wiki/Memoization`

[^2]: In Python 3.0, type `long` is gone; all integers, even really big
    ones, are type `int`.

[^3]: `wikipedia.org/wiki/RSA`

[^4]: `www.cartalk.com/content/puzzler/transcripts/200717`


# Think Python/Tuples

## Tuples are immutable

A tuple is a sequence of values. The values can be any type, and they
are indexed by integers, so in that respect tuples are a lot like lists.
The important difference is that tuples are immutable.

Syntactically, a tuple is a comma-separated list of values:

``` python
>>> t = 'a', 'b', 'c', 'd', 'e'
```

Although it is not necessary, it is common to enclose tuples in
parentheses:

``` python
>>> t = ('a', 'b', 'c', 'd', 'e')
```

To create a tuple with a single element, you have to include the final
comma:

``` python
>>> t1 = ('a',)
>>> type(t1)
<type 'tuple'>
```

Without the comma, Python treats `('a')` as a string in parentheses:

``` python
>>> t2 = ('a')
>>> type(t2)
<type 'str'>
```

Another way to create a tuple is the built-in function `tuple`. With no
argument, it creates an empty tuple:

``` python
>>> t = tuple()
>>> print t
()
```

If the argument is a sequence (string, list or tuple), the result is a
tuple with the elements of the sequence:

``` python
>>> t = tuple('lupins')
>>> print t
('l', 'u', 'p', 'i', 'n', 's')
```

Because `tuple` is the name of a built-in function, you should avoid
using it as a variable name.

Most list operators also work on tuples. The bracket operator indexes an
element:

``` python
>>> t = ('a', 'b', 'c', 'd', 'e')
>>> print t[0]
'a'
```

And the slice operator selects a range of elements.

``` python
>>> print t[1:3]
('b', 'c')
```

But if you try to modify one of the elements of the tuple, you get an
error:

``` python
>>> t[0] = 'A'
TypeError: object doesn't support item assignment
```

You can't modify the elements of a tuple, but you can replace one tuple
with another:

``` python
>>> t = ('A',) + t[1:]
>>> print t
('A', 'b', 'c', 'd', 'e')
```

## Tuple assignment

It is often useful to swap the values of two variables. With
conventional assignments, you have to use a temporary variable. For
example, to swap `a` and `b`:

``` python
>>> temp = a
>>> a = b
>>> b = temp
```

This solution is cumbersome; **tuple assignment** is more elegant:

``` python
>>> a, b = b, a
```

The left side is a tuple of variables; the right side is a tuple of
expressions. Each value is assigned to its respective variable. All the
expressions on the right side are evaluated before any of the
assignments.

The number of variables on the left and the number of values on the
right have to be the same:

``` python
>>> a, b = 1, 2, 3
ValueError: too many values to unpack
```

More generally, the right side can be any kind of sequence (string, list
or tuple). For example, to split an email address into a user name and a
domain, you could write:

``` python
>>> addr = 'monty@python.org'
>>> uname, domain = addr.split('@')
```

The return value from `split` is a list with two elements; the first
element is assigned to `uname`, the second to `domain`.

``` python
>>> print uname
monty
>>> print domain
python.org
```

## Tuples as return values

Strictly speaking, a function can only return one value, but if the
value is a tuple, the effect is the same as returning multiple values.
For example, if you want to divide two integers and compute the quotient
and remainder, it is inefficient to compute `x/y` and then `x%y`. It is
better to compute them both at the same time.

The built-in function `divmod` takes two arguments and returns a tuple
of two values, the quotient and remainder. You can store the result as a
tuple:

``` python
>>> t = divmod(7, 3)
>>> print t
(2, 1)
```

Or use tuple assignment to store the elements separately:

``` python
>>> quot, rem = divmod(7, 3)
>>> print quot
2
>>> print rem
1
```

Here is an example of a function that returns a tuple:

``` python
def min_max(t):
    return min(t), max(t)
```

`max` and `min` are built-in functions that find the largest and
smallest elements of a sequence. `min_max` computes both and returns a
tuple of two values.

## Variable-length argument tuples

Functions can take a variable number of arguments. A parameter name that
begins with `*` **gathers** arguments into a tuple. For example,
`printall` takes any number of arguments and prints them:

``` python
def printall(*args):
    print args
```

The gather parameter can have any name you like, but `args` is
conventional. Here's how the function works:

``` python
>>> printall(1, 2.0, '3')
(1, 2.0, '3')
```

You can combine the gather operator with required and positional
arguments:

``` python
def pointless(required, optional=0, *args):
    print required, optional, args
```

Run this function with 1, 2, 3 and 4 or more arguments and make sure you
understand what it does.

The complement of gather is **scatter**. If you have a sequence of
values and you want to pass it to a function as multiple arguments, you
can use the `*` operator. For example, `divmod` takes exactly two
arguments; it doesn't work with a tuple:

``` python
>>> t = (7, 3)
>>> divmod(t)
TypeError: divmod expected 2 arguments, got 1
```

But if you scatter the tuple, it works:

``` python
>>> divmod(*t)
(2, 1)
```

### Exercise 1

Many of the built-in functions use variable-length argument tuples. For
example, **\'`max`**\' and **\'`min`**\' can take any number of
arguments:\'\'

``` python
''>>> max(1,2,3)
3
''
```

*But **\'`sum`**\' does not.*

``` python
''>>> sum(1,2,3)
TypeError: sum expected at most 2 arguments, got 3
''
```

*Write a function called **\'`sumall`**\' that takes any number of
arguments and returns their sum.*

## Lists and tuples

`zip` is a built-in function that takes two or more sequences and "zips"
them into a list^1^ of tuples where each tuple contains one element from
each sequence.

This example zips a string and a list:

``` python
>>> s = 'abc'
>>> t = [0, 1, 2]
>>> zip(s, t)
[('a', 0), ('b', 1), ('c', 2)]
```

The result is a list of tuples where each tuple contains a character
from the string and the corresponding element from the list.

If the sequences are not the same length, the result has the length of
the shorter one.

``` python
>>> zip('Anne', 'Elk')
[('A', 'E'), ('n', 'l'), ('n', 'k')]
```

You can use tuple assignment in a `for` loop to traverse a list of
tuples:

``` python
t = [('a', 0), ('b', 1), ('c', 2)]
for letter, number in t:
    print number, letter
```

Each time through the loop, Python selects the next tuple in the list
and assigns the elements to `letter` and `number`. The output of this
loop is:

``` python
0 a
1 b
2 c
```

If you combine `zip`, `for` and tuple assignment, you get a useful idiom
for traversing two (or more) sequences at the same time. For example,
`has_match` takes two sequences, `t1` and `t2`, and returns `True` if
there is an index `i` such that `t1[i] == t2[i]`:

``` python
def has_match(t1, t2):
    for x, y in zip(t1, t2):
        if x == y:
            return True
    return False
```

If you need to traverse the elements of a sequence and their indices,
you can use the built-in function `enumerate`:

``` python
for index, element in enumerate('abc'):
    print index, element
```

The output of this loop is:

``` python
0 a
1 b
2 c
```

Again.

## Dictionaries and tuples

Dictionaries have a method called `items` that returns a list of tuples,
where each tuple is a key-value pair^2^.

``` python
>>> d = {'a':0, 'b':1, 'c':2}
>>> t = d.items()
>>> print t
[('a', 0), ('c', 2), ('b', 1)]
```

As you should expect from a dictionary, the items are in no particular
order.

Conversely, you can use a list of tuples to initialize a new dictionary:

``` python
>>> t = [('a', 0), ('c', 2), ('b', 1)]
>>> d = dict(t)
>>> print d
{'a': 0, 'c': 2, 'b': 1}
```

Combining `dict` with `zip` yields a concise way to create a dictionary:

``` python
>>> d = dict(zip('abc', range(3)))
>>> print d
{'a': 0, 'c': 2, 'b': 1}
```

The dictionary method `update` also takes a list of tuples and adds
them, as key-value pairs, to an existing dictionary.

Combining `items`, tuple assignment and `for`, you get the idiom for
traversing the keys and values of a dictionary:

``` python
for key, val in d.items():
    print val, key
```

The output of this loop is:

``` python
0 a
2 c
1 b
```

Again.

It is common to use tuples as keys in dictionaries (primarily because
you can't use lists). For example, a telephone directory might map from
last-name, first-name pairs to telephone numbers. Assuming that we have
defined `last`, `first` and `number`, we could write:

``` python
directory[last,first] = number
```

The expression in brackets is a tuple. We could use tuple assignment to
traverse this dictionary.

``` python
for last, first in directory:
    print first, last, directory[last,first]
```

This loop traverses the keys in `directory`, which are tuples. It
assigns the elements of each tuple to `last` and `first`, then prints
the name and corresponding telephone number.

There are two ways to represent tuples in a state diagram. The more
detailed version shows the indices and elements just as they appear in a
list. For example, the tuple `('Cleese', 'John')` would appear:

```{=html}
<DIV CLASS="center">
```
`<IMG SRC="book020.png">`{=html}

```{=html}
</DIV>
```
But in a larger diagram you might want to leave out the details. For
example, a diagram of the telephone directory might appear:

```{=html}
<DIV CLASS="center">
```
`<IMG SRC="book021.png">`{=html}

```{=html}
</DIV>
```
Here the tuples are shown using Python syntax as a graphical shorthand.

The telephone number in the diagram is the complaints line for the BBC,
so please don't call it.

## Comparing tuples

The comparison operators work with tuples and other sequences; Python
starts by comparing the first element from each sequence. If they are
equal, it goes on to the next elements, and so on, until it finds
elements that differ. Subsequent elements are not considered (even if
they are really big).

``` python
>>> (0, 1, 2) < (0, 3, 4)
True
>>> (0, 1, 2000000) < (0, 3, 4)
True
```

The `sort` function works the same way. It sorts primarily by first
element, but in the case of a tie, it sorts by second element, and so
on.

This feature lends itself to a pattern called **DSU** for

```{=html}
<DL CLASS="description">
```
```{=html}
<DT CLASS="dt-description">
```
**Decorate**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
a sequence by building a list of tuples with one or more sort keys
preceding the elements from the sequence,

```{=html}
</DD>
```
```{=html}
<DT CLASS="dt-description">
```
**Sort**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
the list of tuples, and

```{=html}
</DD>
```
```{=html}
<DT CLASS="dt-description">
```
**Undecorate**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
by extracting the sorted elements of the sequence.

```{=html}
</DD>
```
```{=html}
</DL>
```
For example, suppose you have a list of words and you want to sort them
from longest to shortest:

``` python
def sort_by_length(words):
    t = []
    for word in words:
       t.append((len(word), word))

    t.sort(reverse=True)

    res = []
    for length, word in t:
        res.append(word)
    return res
```

The first loop builds a list of tuples, where each tuple is a word
preceded by its length.

`sort` compares the first element, length, first, and only considers the
second element to break ties. The keyword argument `reverse=True` tells
`sort` to go in decreasing order.

The second loop traverses the list of tuples and builds a list of words
in descending order of length.

### Exercise 2

In this example, ties are broken by comparing words, so words with the
same length appear in alphabetical order. For other applications you
might want to break ties at random. Modify this example so that words
with the same length appear in random order. Hint: see the
**\'`random`**\' function in the **\'`random`**\' module.\'\'

## Sequences of sequences

I have focused on lists of tuples, but almost all of the examples in
this chapter also work with lists of lists, tuples of tuples, and tuples
of lists. To avoid enumerating the possible combinations, it is
sometimes easier to talk about sequences of sequences.

In many contexts, the different kinds of sequences (strings, lists and
tuples) can be used interchangeably. So how and why do you choose one
over the others?

To start with the obvious, strings are more limited than other sequences
because the elements have to be characters. They are also immutable. If
you need the ability to change the characters in a string (as opposed to
creating a new string), you might want to use a list of characters
instead.

Lists are more common than tuples, mostly because they are mutable. But
there are a few cases where you might prefer tuples:

-   In some contexts, like a `return` statement, it is syntactically
    simpler to create a tuple than a list. In other contexts, you might
    prefer a list.
-   If you want to use a sequence as a dictionary key, you have to use
    an immutable type like a tuple or string.
-   If you are passing a sequence as an argument to a function, using
    tuples reduces the potential for unexpected behavior due to
    aliasing.

Because tuples are immutable, they don't provide methods like `sort` and
`reverse`, which modify existing lists. But Python provides the built-in
functions `sorted` and `reversed`, which take any sequence as a
parameter and return a new list with the same elements in a different
order.

## Debugging

Lists, dictionaries and tuples are known generically as **data
structures**; in this chapter we are starting to see compound data
structures, like lists of tuples, and dictionaries that contain tuples
as keys and lists as values. Compound data structures are useful, but
they are prone to what I call **shape errors**; that is, errors caused
when a data structure has the wrong type, size or composition. For
example, if you are expecting a list with one integer and I give you a
plain old integer (not in a list), it won't work.

To help debug these kinds of errors, I have written a module called
`structshape` that provides a function, also called `structshape`, that
takes any kind of data structure as an argument and returns a string
that summarizes its shape. You can download it from
`thinkpython.com/code/structshape.py`

Here's the result for a simple list:

``` python
>>> from structshape import structshape
>>> t = [1,2,3]
>>> print structshape(t)
list of 3 int
```

A fancier program might write "list of 3 int*s*," but it was easier not
to deal with plurals. Here's a list of lists:

``` python
>>> t2 = [[1,2], [3,4], [5,6]]
>>> print structshape(t2)
list of 3 list of 2 int
```

If the elements of the list are not the same type, `structshape` groups
them, in order, by type:

``` python
>>> t3 = [1, 2, 3, 4.0, '5', '6', [7], [8], 9]
>>> print structshape(t3)
list of (3 int, float, 2 str, 2 list of int, int)
```

Here's a list of tuples:

``` python
>>> s = 'abc'
>>> lt = zip(t, s)
>>> print structshape(lt)
list of 3 tuple of (int, str)
```

And here's a dictionary with 3 items that map integers to strings.

``` python
>>> d = dict(lt)
>>> print structshape(d)
dict of 3 int->str
```

If you are having trouble keeping track of your data structures,
`structshape` can help.

## Glossary

```{=html}
<DL CLASS="description">
```
```{=html}
<DT CLASS="dt-description">
```
**tuple:**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
An immutable sequence of elements.

```{=html}
</DD>
```
```{=html}
<DT CLASS="dt-description">
```
**tuple assignment:**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
An assignment with a sequence on the right side and a tuple of variables
on the left. The right side is evaluated and then its elements are
assigned to the variables on the left.

```{=html}
</DD>
```
```{=html}
<DT CLASS="dt-description">
```
**gather:**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
The operation of assembling a variable-length argument tuple.

```{=html}
</DD>
```
```{=html}
<DT CLASS="dt-description">
```
**scatter:**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
The operation of treating a sequence as a list of arguments.

```{=html}
</DD>
```
```{=html}
<DT CLASS="dt-description">
```
**DSU:**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
Abbreviation of "decorate-sort-undecorate," a pattern that involves
building a list of tuples, sorting, and extracting part of the result.

```{=html}
</DD>
```
```{=html}
<DT CLASS="dt-description">
```
**data structure:**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
A collection of related values, often organized in lists, dictionaries,
tuples, etc.

```{=html}
</DD>
```
```{=html}
<DT CLASS="dt-description">
```
**shape (of a data structure):**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
A summary of the type, size and composition of a data structure.

```{=html}
</DD>
```
```{=html}
</DL>
```
## Exercises

### Exercise 3

Write a function called **`most_frequent`** that takes a string and
prints the letters in decreasing order of frequency. Find text samples
from several different languages and see how letter frequency varies
between languages. Compare your results with the tables at
**\'`wikipedia.org/wiki/Letter_frequencies`**\'.\'\'

### Exercise 4

*More anagrams!*

-   \'\'Write a program

that reads a word list from a file (see Section **\'9.1**\') and prints
all the sets of words that are anagrams.Here is an example of what the
output might look like:\'\'

``` python
''['deltas', 'desalt', 'lasted', 'salted', 'slated', 'staled']
['retainers', 'ternaries']
['generating', 'greatening']
['resmelts', 'smelters', 'termless']
''
```

*Hint: you might want to build a dictionary that maps from a set of
letters to a list of words that can be spelled with those letters. The
question is, how can you represent the set of letters in a way that can
be used as a key?*

-   \'\'Modify the previous program so that it prints the largest set

of anagrams first, followed by the second largest set, and so on. \'\'

-   \'\'In Scrabble a "bingo" is when you play all seven tiles in

your rack, along with a letter on the board, to form an eight-letter
word. What set of 8 letters forms the most possible bingos? Hint: there
are seven.\'\'

-   \'\'\'\'Two words form a "metathesis pair" if you can transform one

into the other by swapping two letters**\'^**\'3**\'^**\'; for example,
"converse" and "conserve." Write a program that finds all of the
metathesis pairs in the dictionary. Hint: don't test all pairs of words,
and don't test all possible swaps.**\'**\'You can download a solution
from \'\'***\'`thinkpython.com/code/anagram_sets.py`*****.**\'

### Exercise 5

*Here's another Car Talk Puzzler*^*4*^*:*

> *What is the longest English word, that remains a valid English word,
> as you remove its letters one at a time?* *Now, letters can be removed
> from either end, or the middle, but you can't rearrange any of the
> letters. Every time you drop a letter, you wind up with another
> English word. If you do that, you're eventually going to wind up with
> one letter and that too is going to be an English word---one that's
> found in the dictionary. I want to know what's the longest word and
> how many letters does it have?*
>
> *I'm going to give you a little modest example: Sprite. Ok? You start
> off with sprite, you take a letter off, one from the interior of the
> word, take the r away, and we're left with the word spite, then we
> take the e off the end, we're left with spit, we take the s off, we're
> left with pit, it, and I.*

*Write a program to find all words that can be reduced in this way, and
then find the longest one.*

*This exercise is a little more challenging than most, so here are some
suggestions:*

-   \'\'You might want to write a function that takes a word and

computes a list of all the words that can be formed by removing one
letter. These are the "children" of the word. \'\'

-   \'\'Recursively, a word is reducible if any of its children

are reducible. As a base case, you can consider the empty string
reducible.\'\'

-   \'\'The wordlist I provided, **\'`words.txt`**\', doesn't

contain single letter words. So you might want to add "I", "a", and the
empty string.\'\'

-   \'\'To improve the performance of your program, you might want

to memoize the words that are known to be reducible.\'\'

*You can see my solution at
**\'`thinkpython.com/code/reducible.py`**\'.*

```{=html}
</DIV>
```
```{=html}
<HR CLASS="footnoterule">
```
```{=html}
<DL CLASS="thefootnotes">
```
```{=html}
<DT CLASS="dt-thefootnotes">
```
1

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-thefootnotes">
```
In Python 3.0, `zip` returns an iterator of tuples, but for most
purposes, an iterator behaves like a list.

```{=html}
</DD>
```
```{=html}
<DT CLASS="dt-thefootnotes">
```
2

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-thefootnotes">
```
This behavior is slightly different in Python 3.0.

```{=html}
</DD>
```
```{=html}
<DT CLASS="dt-thefootnotes">
```
3

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-thefootnotes">
```
This exercise is inspired by an example at `puzzlers.org`.

```{=html}
</DD>
```
```{=html}
<DT CLASS="dt-thefootnotes">
```
4

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-thefootnotes">
```
`www.cartalk.com/content/puzzler/transcripts/200651`

```{=html}
</DD>
```
```{=html}
</DL>
```


# Think Python/Case study: data structure selection

## Word frequency analysis

As usual, you should at least attempt the following exercises before you
read my solutions.

### Exercise 1

Write a program that reads a file, breaks each line into words, strips
whitespace and punctuation from the words, and converts them to
lowercase.

*Hint: The **\'`string`**\' module provides strings named
**\'`whitespace`**\', which contains space, tab, newline, etc., and
**\'`punctuation`**\' which contains the punctuation characters. Let's
see if we can make Python swear:*

``` python
>>> import string
>>> print string.punctuation
!"#$%&'()*+,-./:;<=>?@[\]^_`{|}~
```

*Also, you might consider using the string methods **\'`strip`**\',
**\'`replace`**\' and **\'`translate`**\'.*

### Exercise 2

*Go to Project Gutenberg (**\'`gutenberg.org`**\') and download your
favorite out-of-copyright book in plain text format.*

*Modify your program from the previous exercise to read the book you
downloaded, skip over the header information at the beginning of the
file, and process the rest of the words as before.*

*Then modify the program to count the total number of words in the book,
and the number of times each word is used.*

*Print the number of different words used in the book. Compare different
books by different authors, written in different eras. Which author uses
the most extensive vocabulary?*

### Exercise 3

Modify the program from the previous exercise to print the 20 most
frequently-used words in the book.

### Exercise 4

Modify the previous program to read a word list (see
Section **\'9.1**\') and then print all the words in the book that are
not in the word list. How many of them are typos? How many of them are
common words that *should* be in the word list, and how many of them are
really obscure?

## Random numbers

Given the same inputs, most computer programs generate the same outputs
every time, so they are said to be **deterministic**. Determinism is
usually a good thing, since we expect the same calculation to yield the
same result. For some applications, though, we want the computer to be
unpredictable. Games are an obvious example, but there are more.

Making a program truly nondeterministic turns out to be not so easy, but
there are ways to make it at least seem nondeterministic. One of them is
to use algorithms that generate **pseudorandom** numbers. Pseudorandom
numbers are not truly random because they are generated by a
deterministic computation, but just by looking at the numbers it is all
but impossible to distinguish them from random.

The `random` module provides functions that generate pseudorandom
numbers (which I will simply call "random" from here on).

The function `random` returns a random float between 0.0 and 1.0
(including 0.0 but not 1.0). Each time you call `random`, you get the
next number in a long series. To see a sample, run this loop:

``` python
import random
for i in range(10):
    x = random.random()
    print x
```

The function `randint` takes parameters `low` and `high` and returns an
integer between `low` and `high` (including both).

``` python
>>> random.randint(5, 10)
5
>>> random.randint(5, 10)
9
```

To choose an element from a sequence at random, you can use `choice`:

``` python
>>> t = [1, 2, 3]
>>> random.choice(t)
2
>>> random.choice(t)
3
```

The `random` module also provides functions to generate random values
from continuous distributions including Gaussian, exponential, gamma,
and a few more.

### Exercise 5

*Write a function named`choose_from_hist`that takes a histogram as
defined in Section **\'11.1**\' and returns a random value from the
histogram, chosen with probability in proportion to frequency. For
example, for this histogram:*

``` python
''>>> t = ['a', 'a', 'b']
>>> h = histogram(t)
>>> print h
{'a': 2, 'b': 1}
''
```

*your function should **\'`’a’`**\' with probability **\'2/3**\'
and***`b`**\'\' with probability **\'1/3**\'.

## Word histogram

Here is a program that reads a file and builds a histogram of the words
in the file:

``` python
import string

def process_file(filename):
    h = dict()
    fp = open(filename)
    for line in fp:
        process_line(line, h)
    return h

def process_line(line, h):
    line = line.replace('-', ' ')

    for word in line.split():
        word = word.strip(string.punctuation + string.whitespace)
        word = word.lower()

        h[word] = h.get(word, 0) + 1

hist = process_file('emma.txt')
```

This program reads `emma.txt`, which contains the text of *Emma* by Jane
Austen.

`process_file` loops through the lines of the file, passing them one at
a time to `process_line`. The histogram `h` is being used as an
accumulator.

`process_line` uses the string method `replace` to replace hyphens with
spaces before using `split` to break the line into a list of strings. It
traverses the list of words and uses `strip` and `lower` to remove
punctuation and convert to lower case. (It is a shorthand to say that
strings are "converted;" remember that string are immutable, so methods
like `strip` and `lower` return new strings.)

Finally, `process_line` updates the histogram by creating a new item or
incrementing an existing one.

To count the total number of words in the file, we can add up the
frequencies in the histogram:

``` python
def total_words(h):
    return sum(h.values())
```

The number of different words is just the number of items in the
dictionary:

``` python
def different_words(h):
    return len(h)
```

Here is some code to print the results:

``` python
print 'Total number of words:', total_words(hist)
print 'Number of different words:', different_words(hist)
```

And the results:

``` python
Total number of words: 161073
Number of different words: 7212
```

## Most common words

To find the most common words, we can apply the DSU pattern;
`most_common` takes a histogram and returns a list of word-frequency
tuples, sorted in reverse order by frequency:

``` python
def most_common(h):
    t = []
    for key, value in h.items():
        t.append((value, key))

    t.sort(reverse=True)
    return t
```

Here is a loop that prints the ten most common words:

``` python
t = most_common(hist)
print 'The most common words are:'
for freq, word in t[0:10]:
    print word, '\t', freq
```

And here are the results from *Emma*:

``` python
The most common words are:
to      5242
the     5204
and     4897
of      4293
i       3191
a       3130
it      2529
her     2483
was     2400
she     2364
```

## Optional parameters

We have seen built-in functions and methods that take a variable number
of arguments. It is possible to write user-defined functions with
optional arguments, too. For example, here is a function that prints the
most common words in a histogram

``` python
def print_most_common(hist, num=10)
    t = most_common(hist)
    print 'The most common words are:'
    for freq, word in t[0:num]:
        print word, '\t', freq
```

The first parameter is required; the second is optional. The **default
value** of `num` is 10.

If you only provide one argument:

``` python
print_most_common(hist)
```

`num` gets the default value. If you provide two arguments:

``` python
print_most_common(hist, 20)
```

`num` gets the value of the argument instead. In other words, the
optional argument **overrides** the default value.

If a function has both required and optional parameters, all the
required parameters have to come first, followed by the optional ones.

## Dictionary subtraction

Finding the words from the book that are not in the word list from
`words.txt` is a problem you might recognize as set subtraction; that
is, we want to find all the words from one set (the words in the book)
that are not in another set (the words in the list).

`subtract` takes dictionaries `d1` and `d2` and returns a new dictionary
that contains all the keys from `d1` that are not in `d2`. Since we
don't really care about the values, we set them all to None.

``` python
def subtract(d1, d2):
    res = dict()
    for key in d1:
        if key not in d2:
            res[key] = None
    return res
```

To find the words in the book that are not in `words.txt`, we can use
`process_file` to build a histogram for `words.txt`, and then subtract:

``` python
words = process_file('words.txt')
diff = subtract(hist, words)

print "The words in the book that aren't in the word list are:"
for word in diff.keys():
    print word,
```

Here are some of the results from *Emma*:

``` python
The words in the book that aren't in the word list are:
 rencontre jane's blanche woodhouses disingenuousness
friend's venice apartment ...
```

Some of these words are names and possessives. Others, like "rencontre,"
are no longer in common use. But a few are common words that should
really be in the list!

### Exercise 6

\'\'Python provides a data structure called **\'`set`**\' that provides
many common set operations. Read the documentation at
**\'`docs.python.org/lib/types-set.html`**\' and write a program that
uses set subtraction to find words in the book that are not in the word
list.

## Random words

To choose a random word from the histogram, the simplest algorithm is to
build a list with multiple copies of each word, according to the
observed frequency, and then choose from the list:

``` python
def random_word(h):
    t = []
    for word, freq in h.items():
        t.extend([word] * freq)

    return random.choice(t)
```

The expression `[word] * freq` creates a list with `freq` copies of the
string `word`. The `extend` method is similar to `append` except that
the argument is a sequence.

### Exercise 7

*This algorithm works, but it is not very efficient; each time you
choose a random word, it rebuilds the list, which is as big as the
original book. An obvious improvement is to build the list once and then
make multiple selections, but the list is still big.*

*An alternative is:*

-   *Use **\'`keys`**\' to get a list of the words in the book.*
-   \'\'Build a list that contains the cumulative sum of the word

frequencies (see Exercise **\'10.1**\'). The last item in this list is
the total number of words in the book, ***\'n***\'.\'\'

-   *Choose a random number from 1 to***\'n**\'\'\'. Use a bisection
    search

(See Exercise **\'10.8**\') to find the index where the random number
would be inserted in the cumulative sum.\'\'

-   *Use the index to find the corresponding word in the word list.*

\'\'Write a program that uses this algorithm to choose a random word
from the book.

## Markov analysis

If you choose words from the book at random, you can get a sense of the
vocabulary, you probably won't get a sentence:

``` python
this the small regard harriet which knightley's it most things
```

A series of random words seldom makes sense because there is no
relationship between successive words. For example, in a real sentence
you would expect an article like "the" to be followed by an adjective or
a noun, and probably not a verb or adverb.

One way to measure these kinds of relationships is Markov analysis,
which characterizes, for a given sequence of words, the probability of
the word that comes next. For example, the song *Eric, the Half a Bee*
begins:

> Half a bee, philosophically,\
> Must, ipso facto, half not be.\
> But half the bee has got to be\
> Vis a vis, its entity. D'you see?\
> \
> But can a bee be said to be\
> Or not to be an entire bee\
> When half the bee is not a bee\
> Due to some ancient injury?\

In this text, the phrase "half the" is always followed by the word
"bee," but the phrase "the bee" might be followed by either "has" or
"is".

The result of Markov analysis is a mapping from each prefix (like "half
the" and "the bee") to all possible suffixes (like "has" and "is").

Given this mapping, you can generate a random text by starting with any
prefix and choosing at random from the possible suffixes. Next, you can
combine the end of the prefix and the new suffix to form the next
prefix, and repeat.

For example, if you start with the prefix "Half a," then the next word
has to be "bee," because the prefix only appears once in the text. The
next prefix is "a bee," so the next suffix might be "philosophically,"
"be" or "due."

In this example the length of the prefix is always two, but you can do
Markov analysis with any prefix length. The length of the prefix is
called the "order" of the analysis.

### Exercise 8

Markov analysis:\'\'

-   *Write a program to read a text from a file and perform Markov
    analysis. The result should be a dictionary that maps from prefixes
    to a collection of possible suffixes. The collection might be a
    list, tuple, or dictionary; it is up to you to make an appropriate
    choice. You can test your program with prefix length two, but you
    should write the program in a way that makes it easy to try other
    lengths.*
-   *Add a function to the previous program to generate random text
    based on the Markov analysis. Here is an example from*Emma\'\' with
    prefix length 2:

> He was very clever, be it sweetness or be angry, ashamed or only
> amused, at such a stroke. She had never thought of Hannah till you
> were never meant for me?\" \"I cannot make speeches, Emma:\" he soon
> cut it all himself.

\'\'For this example, I left the punctuation attached to the words. The
result is almost syntactically correct, but not quite. Semantically, it
almost makes sense, but not quite.

-   *What happens if you increase the prefix length? Does the random
    text make more sense?*
-   *Once your program is working, you might want to try a mash-up: if
    you analyze text from two or more books, the random text you
    generate will blend the vocabulary and phrases from the sources in
    interesting ways.*

## Data structures

Using Markov analysis to generate random text is fun, but there is also
a point to this exercise: data structure selection. In your solution to
the previous exercises, you had to choose:

-   How to represent the prefixes.
-   How to represent the collection of possible suffixes.
-   How to represent the mapping from each prefix to the collection of
    possible suffixes.

Ok, the last one is the easy; the only mapping type we have seen is a
dictionary, so it is the natural choice.

For the prefixes, the most obvious options are string, list of strings,
or tuple of strings. For the suffixes, one option is a list; another is
a histogram (dictionary).

How should you choose? The first step is to think about the operations
you will need to implement for each data structure. For the prefixes, we
need to be able to remove words from the beginning and add to the end.
For example, if the current prefix is "Half a," and the next word is
"bee," you need to be able to form the next prefix, "a bee."

Your first choice might be a list, since it is easy to add and remove
elements, but we also need to be able to use the prefixes as keys in a
dictionary, so that rules out lists. With tuples, you can't append or
remove, but you can use the addition operator to form a new tuple:

``` python
def shift(prefix, word):
    return prefix[1:] + (word,)
```

`shift` takes a tuple of words, `prefix`, and a string, `word`, and
forms a new tuple that has all the words in `prefix` except the first,
and `word` added to the end.

For the collection of suffixes, the operations we need to perform
include adding a new suffix (or increasing the frequency of an existing
one), and choosing a random suffix.

Adding a new suffix is equally easy for the list implementation or the
histogram. Choosing a random element from a list is easy; choosing from
a histogram is harder to do efficiently (see Exercise 13.7).

So far we have been talking mostly about ease of implementation, but
there are other factors to consider in choosing data structures. One is
run time. Sometimes there is a theoretical reason to expect one data
structure to be faster than other; for example, I mentioned that the
`in` operator is faster for dictionaries than for lists, at least when
the number of elements is large.

But often you don't know ahead of time which implementation will be
faster. One option is to implement both of them and see which is better.
This approach is called **benchmarking**. A practical alternative is to
choose the data structure that is easiest to implement, and then see if
it is fast enough for the intended application. If so, there is no need
to go on. If not, there are tools, like the `profile` module, that can
identify the places in a program that take the most time.

The other factor to consider is storage space. For example, using a
histogram for the collection of suffixes might take less space because
you only have to store each word once, no matter how many times it
appears in the text. In some cases, saving space can also make your
program run faster, and in the extreme, your program might not run at
all if you run out of memory. But for many applications, space is a
secondary consideration after run time.

One final thought: in this discussion, I have implied that we should use
one data structure for both analysis and generation. But since these are
separate phases, it would also be possible to use one structure for
analysis and then convert to another structure for generation. This
would be a net win if the time saved during generation exceeded the time
spent in conversion.

## Debugging

When you are debugging a program, and especially if you are working on a
hard bug, there are four things to try:

```{=html}
<DL CLASS="description">
```
```{=html}
<DT CLASS="dt-description">
```
**reading:**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
Examine your code, read it back to yourself, and check that it says what
you meant to say.

```{=html}
</DD>
```
```{=html}
<DT CLASS="dt-description">
```
**running:**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
Experiment by making changes and running different versions. Often if
you display the right thing at the right place in the program, the
problem becomes obvious, but sometimes you have to spend some time to
build scaffolding.

```{=html}
</DD>
```
```{=html}
<DT CLASS="dt-description">
```
**ruminating:**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
Take some time to think! What kind of error is it: syntax, runtime,
semantic? What information can you get from the error messages, or from
the output of the program? What kind of error could cause the problem
you're seeing? What did you change last, before the problem appeared?

```{=html}
</DD>
```
```{=html}
<DT CLASS="dt-description">
```
**retreating:**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
At some point, the best thing to do is back off, undoing recent changes,
until you get back to a program that works and that you understand. Then
you can starting rebuilding.

```{=html}
</DD>
```
```{=html}
</DL>
```
Beginning programmers sometimes get stuck on one of these activities and
forget the others. Each activity comes with its own failure mode.

For example, reading your code might help if the problem is a
typographical error, but not if the problem is a conceptual
misunderstanding. If you don't understand what your program does, you
can read it 100 times and never see the error, because the error is in
your head.

Running experiments can help, especially if you run small, simple tests.
But if you run experiments without thinking or reading your code, you
might fall into a pattern I call "random walk programming," which is the
process of making random changes until the program does the right thing.
Needless to say, random walk programming can take a long time.

You have to take time to think. Debugging is like an experimental
science. You should have at least one hypothesis about what the problem
is. If there are two or more possibilities, try to think of a test that
would eliminate one of them.

Taking a break helps with the thinking. So does talking. If you explain
the problem to someone else (or even yourself), you will sometimes find
the answer before you finish asking the question.

But even the best debugging techniques will fail if there are too many
errors, or if the code you are trying to fix is too big and complicated.
Sometimes the best option is to retreat, simplifying the program until
you get to something that works and that you understand.

Beginning programmers are often reluctant to retreat because they can't
stand to delete a line of code (even if it's wrong). If it makes you
feel better, copy your program into another file before you start
stripping it down. Then you can paste the pieces back in a little bit at
a time.

Finding a hard bug requires reading, running, ruminating, and sometimes
retreating. If you get stuck on one of these activities, try the others.

## Glossary

```{=html}
<DL CLASS="description">
```
```{=html}
<DT CLASS="dt-description">
```
**deterministic:**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
Pertaining to a program that does the same thing each time it runs,
given the same inputs.

```{=html}
</DD>
```
```{=html}
<DT CLASS="dt-description">
```
**pseudorandom:**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
Pertaining to a sequence of numbers that appear to be random, but are
generated by a deterministic program.

```{=html}
</DD>
```
```{=html}
<DT CLASS="dt-description">
```
**default value:**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
The value given to an optional parameter if no argument is provided.

```{=html}
</DD>
```
```{=html}
<DT CLASS="dt-description">
```
**override:**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
To replace a default value with an argument.

```{=html}
</DD>
```
```{=html}
<DT CLASS="dt-description">
```
**benchmarking:**

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-description">
```
The process of choosing between data structures by implementing
alternatives and testing them on a sample of the possible inputs.

```{=html}
</DD>
```
```{=html}
</DL>
```
## Exercises

### Exercise 9

*The "rank" of a word is its position in a list of words sorted by
frequency: the most common word has rank 1, the second most common has
rank 2, etc.*

*Zipf's law describes a relationship between the ranks and frequencies
of words in natural languages*^*1*^*. Specifically, it predicts that the
frequency,***\'f, of the word with rankr***\' is:*

```{=html}
<TABLE CLASS="display dcenter">
```
```{=html}
<TR VALIGN="middle">
```
```{=html}
<TD CLASS="dcell">
```
**\'f* = *c* *r**\'`<SUP>`{=html}*−*s\'*\'`</SUP>`{=html}* \'\'

```{=html}
</TD>
```
```{=html}
</TR>
```
```{=html}
</TABLE>
```
*where***\'sandc***\' are parameters that depend on the language and the
text. If you take the logarithm of both sides of this equation, you
get:*

```{=html}
<TABLE CLASS="display dcenter">
```
```{=html}
<TR VALIGN="middle">
```
```{=html}
<TD CLASS="dcell">
```
*log*\'*\'f* = **\'log***\'c* − *s* **\'log***\'r* \'\'

```{=html}
</TD>
```
```{=html}
</TR>
```
```{=html}
</TABLE>
```
*So if you plot **\'logfversus**\'log***\'r, you should get a straight
line with slope**\'−*s***\' and intercept**\'log***\'c***\'.\'\'

*Write a program that reads a text from a file, counts word frequencies,
and prints one line for each word, in descending order of frequency,
with **\'logfand**\'log***\'r. Use the graphing program of your choice
to plot the results and check whether they form a straight line. Can you
estimate the value of***s\'?*

## Notes

```{=html}
<DL CLASS="thefootnotes">
```
```{=html}
<DT CLASS="dt-thefootnotes">
```
1

```{=html}
</DT>
```
```{=html}
<DD CLASS="dd-thefootnotes">
```
See `wikipedia.org/wiki/Zipf's_law`

```{=html}
</DD>
```
```{=html}
</DL>
```